Advanced intelligence engine

ABSTRACT

An advanced intelligence engine (AIE) for use in identifying what may be complex events or developments on one or more data platforms or networks from various types of structured or normalized data generated by one or more disparate data sources. The AIE may conduct one or more types of quantitative, correlative, behavioral and corroborative analyses to detect events from what may otherwise be considered unimportant or non-relevant information spanning one or more time periods. Events generated by the AIE may be passed to an event manager to determine whether further action is required such as reporting, remediation, and the like.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/026,834, entitled “ADVANCED INTELLIGENCE ENGINE,” filed on Sep. 13,2013, which is a continuation of U.S. patent application Ser. No.13/303,526 entitled “ADVANCED INTELLIGENCE ENGINE,” filed on Nov. 23,2011, and now U.S. Pat. No. 8,543,694, which claims priority to U.S.Provisional Application No. 61/417,114 entitled “ADVANCED INTELLIGENCEENGINE,” filed on Nov. 24, 2010. The entire contents of theaforementioned applications are hereby incorporated within by referenceas if set forth in full.

FIELD OF THE INVENTION

The present invention relates in general to network monitoring andinformation management that allows for event detection and analysisbased on the processing and organization of log messages and other typesof structured or normalized data.

BACKGROUND

Modern business operations typically require many communication devicesand technologies that include routers, firewalls, switches, fileservers, ERP applications, etc. Generally, such devices and technologiesreport their health and status by writing log files. For example,computer processors are responsible for processing vast amounts of datafor a variety of applications. To determine how a certain applicationmay be processed by a computer processor, engineers typically design theapplication with a log file that records various functional outputswithin the application. That is, certain functions within theapplication may output data to the log file so that the engineers maydiagnose problems (e.g., software bugs) and/or observe generaloperational characteristics of the application.

By observing the general operational characteristics of an application,certain valuable information may also be ascertained. For example, logfiles generated by a file server may record logins. In this regard,certain logins may be unauthorized and their prevention desired.However, with the multitude of communication devices and theircorresponding applications available, a bewildering array of log datamay be generated within a communication network. Additionally,communication networks are often upgraded with additional systems thatprovide even more logs. Adding to the complexity of the situation,communication devices and applications of these communication networksvary in many ways and so do their corresponding log file formats.

Attempts have been made by network and enterprise administrators toextract or identify useful information from the large volume of logmessages generated by the aforementioned communication devices andtechnologies via one or more types of Security Event Management (SEM)solutions. Some of these attempts involve processing logs against one ormore rules in an attempt to identify “events” that may be furtheranalyzed by administrators and troubleshooters. Events may be anyoccurrence (as identified by one or more log messages) that may be ofinterest to a network administrator for further analysis. Examplesinclude one or more particular IP address attempting to access aparticular network device, a large quantity of data leaving the network,etc. For instance, upon at least a portion of a log message matching arule, one or more particular actions may take place (e.g., archiving ofthe log message, alerting an administrator of the occurrence of theevent, etc.).

SUMMARY OF THE INVENTION

The inventors have determined that traditional SEM solutions have anumber of fundamental failings when it comes to detecting sophisticatedintrusions, insider threats, and the like. In one regard, thesetraditional SEM solutions are often limited to logs that are alreadydesignated as “events.” For instance, some traditional SEM solutionsdepend upon a monitored device pre-filtering log messages for allegedlyimportant logs messages and then passing such messages to a processingengine implementing the traditional SEM solutions. Traditional SEMsolutions have not been architected or designed to collect and analyzeall log data regardless of source. Stated differently, these solutionshave generally been designed with a limited scope in mind which hasinherently limited their current and future analysis capabilities.

In another regard, traditional SEMs have confined themselves to the oncethought “holy grail” of security—“correlation” (i.e., processing logmessages against rules defining a known pattern of activity, where thatpattern of activity may span multiple log messages from differentdevices). While correlation has at times proved useful, only things wellknown and understood can be correlated (e.g., knowing any attemptedaccess of a network by a particular IP address is always cause forconcern); thus, correlation is often limited based on analysis scope andcomplexity. Furthermore, each of researching, understanding, anddeveloping these pattern-based rules is inherently complex which furtherlimits overall usefulness.

With these shortcomings of traditional SEM solutions in mind, theinventors have determined that systems, apparatuses, processes, methods,etc. (“utilities”) are needed that can collect and analyze most or alllog or other types of structured or normalized data (e.g.,transactional, activity, etc.) spread across multiple systems, devices,applications and databases and spanning any number of time periods byleveraging a combination of analysis techniques that independently offervalue but, when combined, deliver a fundamental breakthrough in thedetection of sophisticated intrusions and computer crime. Stateddifferently, utilities are needed that can, for instance, receive,process and extract what may otherwise be considered unimportant ornon-relevant information from disparate data sources, but, whenconsidered together (even over disparate time periods), may indicate aparticular event or even an alarm that can be further analyzed and/oraddressed by appropriate personnel.

Imagine the situation where a device originating from an externalinternet protocol (IP) address is launching an attack on a particularnetwork by attempting to disrupt, deny, degrade, or destroy informationresident on the network. For instance, the external device could attemptto send a code or instruction to a central processing unit (CPU) of aserver within the particular network for causing the server to beginpropagating malware throughout the network. A traditional SEM solutioncould detect the above event by, inter alia, utilizing any appropriatelog processing rule(s) specifically designed to isolate log messagesthat originate from the external IP address (assuming the external IPaddress is known). In response to detecting that the external IP addressis attempting a network attack, the log processing rule may be designedto send a message to a network administrator indicative of the attack.

However, what if the message to the network administrator indicative ofthe attack is never sent? As further examples, what if a backup of anetwork server or device started but never finished, or what if a serverwas attacked and later started transferring large amounts of data out ofthe network? Traditional SEM solutions have not been designed toautomatically and seamlessly detect interesting instances orobservations (or non-observations) that are linked or related to one ormore other occurrences (e.g., such as the above network attack) and thatare indicative of, for instance, a transfer of a large amount of dataout of a network in a particular time period after a network device wascontacted by an external IP address, a failure to send a message to anetwork administrator indicative of an attack, a failure to properlycomplete a backup of a network server, and the like.

As will be discussed herein, the present utilities serve to provide asingle platform that allows administrators, troubleshooters, and otherusers to perform various types of analyses of structured data (e.g.,log, transactional, activity) such as correlative, behavioral,statistical, corroborative, and the like. More specifically, the presentutilities can be customized (e.g., in conjunction with a console or userinterface) to dynamically detect a wide range of various combinations ofnetwork and other occurrences, such as the above example, in a mannerthat eliminates or limits the weaknesses in traditional notions ofstructured data-monitoring and detection (e.g., such as eventcorrelation). Among other abilities, the present utilities can process asignificant volume of structured or normalized data spread acrossmultiple systems and devices in a memory and CPU efficient manner andirrespective of a time in which data is processed to extract or gleannumerous types of useful information from the processed data (e.g.,detecting sophisticated intrusions following known patterns and insiderthreats that may be effectively “invisible” to the organization). Theanalyses may be performed in real time to identify suspicious activityand known issues as they occur by providing a scalable solution that iscapable of indexing and analyzing significant quantities of structureddata and providing various analysis technique such as correlativeanalyses (e.g., generating an event if events a, b or c are observedfollowed by event d in timeframe z), statistical analyses (e.g.,creating an event if data sent exceeds quantity a in timeframe z),behavioral analyses (e.g., creating an event if a user is observedauthenticating to a network during timeframes that are significantlydifferent than previously established usage patterns) corroborativeanalyses (e.g., creating an event if an attack is observed against asystem known to be vulnerable to a), and the like.

Generally, the present utilities make use of one or more “rule blocks”or “rule modules,” each of which is operable to obtain and store one ormore types of “facts” (e.g., log related or other types of structureddata), determine whether one or more “conditions” related to the factshave been satisfied (e.g., the observation or non-observation ofparticular types of facts, exceeding or not exceeding a threshold volumeof transferred data, and/or the like), and take one or more types ofactions upon satisfaction of the one or more conditions. One or morerule blocks can make up a log processing, structured data, or “AdvancedIntelligence Engine” (AIE) rule. One or more “linking relationships” canconnect adjacent rule blocks, each of which generally describes how onerule block is related to an adjacent rule block (e.g., a destination IPaddress in a first rule block is equal to a source IP address in asecond rule block). Upon satisfaction of the condition(s) of one ruleblock (e.g., upon “triggering” or “firing” of a rule block) of an AIErule (sometimes regardless of an order of the rule block relative toother rule blocks in the AIE rule), each of the one or more other ruleblocks of the AIE rule is automatically evaluated by way of the variouslinking relationships to determine if its conditions has also beensatisfied. When the conditions of all of the one or more rule blocks ofan AIE rule have been satisfied, an “event” is generated that isindicative of satisfaction of all the rule blocks and which may bestored for use by personnel or the like.

For instance, a network administrator may be interested in knowing aboutany large data transfers that occur from a internal network device to anexternal IP address in a 30 minute period after the external IP addresshas accessed the internal network device, as such a combination ofoccurrences may indicate that a hacker has accessed the network andbegun obtaining sensitive data from the network. In this regard, theadministrator may design a first rule block that is configured toobtain, store and/or track all log messages generated by internalnetwork devices as a function of time, and monitor for any connectionsbetween an external IP address and an internal IP address (e.g., wherethe first rule block's condition would be satisfied upon observing sucha connection). For instance, upon the first rule block determining thata newly received log message includes an external source IP address andan internal destination IP address, the first rule block may “fire” orotherwise consider that its condition has been satisfied. That is,firing of the rule block upon or after an evaluation may be considered afirst or desired/interesting outcome while non-firing of the rule blockupon or after an evaluation may be considered a second ornon-desired/non-interesting outcome.

The administrator may also design a second rule block that is configuredto obtain, store and/or track log messages indicative of all outbounddata transfers as a function of time, and monitor for any outbound datatransfer from an internal device that is at least 1 GB in size. As partof creating an AIE rule, the administrator may then configure at leastone linking relationship between the first and second rule blocks. Thelinking relationship may specify the particular data field content thatis to be used, upon firing of one rule block, as a key into an indexstructure associated with the data of the other rule block. The linkingrelationship may also specify the temporal relationship between whenadjacent rule block conditions must have been satisfied in order for anevent to be generated. For instance, the administrator may equate thedestination IP address field content of the first rule block to thesource IP address field content of the second rule block. Theadministrator may also specify that an event is to be generated only ifthe second rule block's condition was satisfied within 30 minutes afterthe connection between the external and internal IP address was made(wherein the specific time of the connection may be different than thespecific time at which the first rule block was or is satisfied, due toprocessing delays and the like).

To illustrate, imagine that upon evaluation of the first rule block(which may occur according to a predetermined schedule and/or upon a newpiece of data being received by the first rule block), it is determinedthat an external/internal connection has occurred and thus that thefirst rule block's condition has been satisfied. Thereafter, the secondrule block may be evaluated to determine if its condition has beensatisfied in the 30 minutes after the external/internal connectionoccurred by using the specific destination IP address of the connectionas a key into an index structure of data of the second rule block (wherethe source IP address of any over 1GB outbound data transfer matches thespecific destination IP address of the connection); an event may begenerated upon a positive determination while no event may be generatedupon a negative determination.

It may also be the case that the second rule block fires before thefirst rule block fires. For instance, upon determination that anoutbound data transfer of 2 GB has occurred from an internal device(e.g., which may be indicated by one or possibly multiple log messages)and thus that the second rule block's condition has been satisfied, thefirst rule block may then be evaluated to determine whether itscondition has been satisfied. In this situation, the specific source IPaddress of the internal device associated with the 2 GB outbound datatransfer would then be used as a key into an index structure of data ofthe first rule block to determine if an external/internal connectionoccurred within the 30 minutes preceding the 2 GB outbound data transfer(where the destination/internal IP address of the connection matches thespecific source IP address of the internal device associated with the 2GB outbound data transfer); an event may be generated upon a positivedetermination while no event may be generated upon a negativedetermination.

The situation where facts indicative of a large outbound data transferby an internal device are processed before facts indicative of anexternal connection to the internal device that precipitated the largeoutbound data transfer highlights how the present utilities canaccommodate “out of time order” data processing while maintainingsuccessful and/or efficient evaluation of an AIE rule. That is,regardless of the particular order in which log messages are processed,any temporal conditions linking adjacent rule blocks (e.g., the 30minute interval in the above example) may be measured in relation toreal-time or in other words according to the time the incident (e.g.,the external/internal connection or the outbound data transfer) actuallyoccurred (e.g., as measured by any appropriate log message time stamp)as opposed to when the rule block was evaluated or when data wasprocessed.

The present utilities may also be used as part of a behavioral and/orrelational analysis of structured or normalized facts. For instance, afirst rule block could be designed to continually filter for orotherwise obtain one or more particular types of facts (e.g., all logdata from a particular source, all log data having a particularclassification, and/or the like). Also, a second rule block could bedesigned to make one or more determinations relating to the factsobtained by the first rule block (i.e., instead of the second rule blockmaking determinations about facts that the second rule block obtains asdiscussed above). In one arrangement, administrators could set up afirst plurality of rule blocks designed to obtain various types offacts, and a second plurality of rule blocks designed to ask variousrespective questions or otherwise make various respective determinationsabout whether specific types of facts exist, whether a threshold countrelated to the facts has been reached (e.g., equal to or over x numberof bytes, equal to or over×log count, etc.), and the like. For instance,administrators can “mix and match” various one of the first plurality ofrule blocks to various ones of the second plurality of rule blocks toallow for customization of various types of analyses to obtain variousevents of interest.

The present utilities may also encompass various techniques to identifywhat may be considered “false alarms.” More specifically, while an AIErule including first and second (and/or additional) rule blocks mayappear to identify an event that typically would be considered“interesting” to an administrator or other personnel, the specific factsgiving rise to the “event” may be subjected to additional processingthat may effectively cause the non-generation of an event. For instance,imagine that an AIE rule includes a first rule block that fires when anexternal/internal connection is made, a second rule block that fireswhen an internal IP address transfers large amounts of data to theexternal address after the connection, and a linking relationship objectthat equates the particular impacted IP address of the first rule blockwith the particular origin IP address of the second rule block. Whilethis AIE rule may have been designed to identify a hacker accessing dataof a network, it may falsely identify a legitimate employee accessingthe network from his or her home computer.

In one arrangement, the false alarm mitigation techniques disclosedherein may subject the facts collected by the rule blocks to variousadditional questions or queries that would tend to corroborate whetheror not an event has actually been identified. For instance, if theparticular IP address identified above was typically associated withlarge data transfers after hours, an event may thus not be generated forthis particular scenario. In further embodiments, a plurality ofadditional questions or queries may be asked or made of the facts, wherea positive answer to at least some of the questions or queries mayeffectively cancel out the generation of an event.

In another arrangement, the false alarm mitigation techniques mayincorporate various types of closed-loop or fine-tuning schemes thatallow the utilities to dynamically adjust various parameters based onhistorical results. For instance, while a particular AIE rule couldinclude a rule block that is designed to fire when a particularthreshold number of a quantitative value of a field (e.g., log count,bytes transferred) has been reached, the threshold number coulddynamically adjust upwardly or downwardly based on historical results.In one embodiment, “indirect” administrator input may cause a thresholdto move up or down, such as a threshold dynamically moving upwardly whenan administrator continually indicates that an “event” generated from anAIE rule including a rule block with a particular threshold is notreally an event. As another example, while one rule block could bedesigned to detect whether a “server backup completion” message has beengenerated in the 3 hours after another rule block has fired (e.g.,design to detect whether a server backup start message has beengenerated), it may be the case that the average time that previous suchserver backup completion messages fire is only 1 hour after the serverbackup start message has been generated. Thus, the above 3 hour timeperiod may dynamically adjust downwardly as appropriate.

According to one aspect, a utility for use in monitoring one or moreplatforms of one or more data systems, includes receiving, at aprocessing engine, structured data generated by one or more platformsover at least one communications network; first evaluating, at theprocessing engine using a first rule block (RB), at least some of thedata; first determining, from the first evaluating, whether a result isone of at least first and second outcomes; and depending upon thedetermining, second evaluating, at the processing engine using a secondRB, at least some of the data; and second determining, from the secondevaluating, whether a result is one of at least first and secondoutcomes, where the results are analyzed to determine an event ofinterest.

In one arrangement, the first determining includes ascertaining that theresult of the first evaluating is the first outcome. For instance, thefirst outcome may be that a “condition” of the rule block has beensatisfied (e.g., a potentially “interesting” occurrence such as a serverbackup has started, an unusual outside IP address attempting to connectwith a local machine, etc.) and the second outcome may be an“uninteresting” occurrence (e.g., a known employee logs into his or herwork email server during business hours).

In any case, the method may further include ascertaining that the resultof the second evaluating is the first outcome (i.e., the first outcomeof the second rule block), and then generating an event after the resultof the second evaluating is determined to be the first outcome. Thesecond rule block could be designed to monitor for an occurrence that isat least somewhat related to the occurrence being monitored in the firstrule block. For instance, the second rule block may be designed tomonitor for a failure to receive a “server backup completion” messagewithin a particular period of time corresponding to the specific IPaddress of the server associated with the server backup process, and thefirst outcome may be that no such completion message was received. Inconnection with generation of an event, an alarm may be generated whichmay be forwarded to any desired entities (e.g., administrators) to allowappropriate remedial action to be taken.

The first outcome of the first evaluating may occur at a first time thatis identified by at least one time stamp of the at least some of thedata (i.e., the data considered and evaluated by the first rule block).That is, the time at which the “condition” of the first rule block (andother rule blocks) is considered “satisfied” may be measured by thespecific time stamp(s) of the data that gave rise to the satisfaction ofthe condition (i.e., instead of the specific time at which the ruleblock determined that its condition was satisfied). In some embodiments,the second evaluating may include considering data of the at least someof the data that is associated with at least one time stamp including asecond time that is the same as or after the first time (i.e., of thefirst rule block), or even before the first time.

Each of the first and second evaluating steps may include making somedetermination about one or more fields of the at least some of the data,and a content of one of the fields in the first evaluating matches acontent of one of the fields in the second evaluating. Stateddifferently, there may be at least one matching piece of content in thedata evaluated by the first and second rule blocks (e.g., such as acommon IP address). For instance, upon the first evaluating resulting inthe first outcome (e.g., a determination that the first rule block'scondition has been satisfied), the matching or common piece of contentmay be used as a key into an index structure of the at least some of thedata (e.g., an index structure of an in-memory data structure maintainedby the second rule block) to determine if the result is one of the firstand second outcomes (e.g., to determine if the second rule block'scondition has been satisfied, and thus whether an event should possiblybe generated).

In one arrangement, the first outcome of at least one of the first andsecond evaluating includes data of the at least some of the datamatching or not matching first criteria. For instance, the firstcriteria may include a particular content of one or more fields of thedata of the at least some of the data. As another example, the firstoutcome may include a particular quantitative value (e.g., log count,bytes transferred, etc.) of the data matching the first criteria beingequal to or exceeding a threshold for the particular quantitative value.As a further example, the first outcome may further include a particularfield of the data comprising at least a threshold number of uniquecontents of the particular field. The particular time period may bemeasured in relation to a time at which the first outcome of one of thefirst and second evaluating occurred.

In some arrangements, further rule blocks could be processed at leastpartially in conjunction with the first and second rule blocks. Forinstance, in the case where the result of each of the first and secondevaluating includes the first outcomes, the method could further includethird evaluating, at the processing engine using a third rule block, atleast some of the data, where the results are analyzed to determine anevent of interest.

In one arrangement, the first and/or second evaluating may includeaggregating the data of at least one common field of the data receivedat the processing engine, determining a total count of the aggregateddata, and utilizing the total count in determining whether or not theresult of the first and/or second evaluating includes the firstcondition or second condition. This arrangement advantageously allowsfor the efficient management of what may be large volumes of structureddata. In another arrangement, additional (e.g., identical) eventsmessages may be suppressed during a suppression window (e.g., for 30minutes after an initial event). Any of the rule blocks may beconfigured to continually receive updated data and reevaluate the datain any appropriate manner (e.g., when new data is received, according toa predefined schedule, etc.).

According to another aspect, an event generating utility includes anevent module that is adapted to be operatively interposed between a userconsole for manipulating the event module and at least one source ofstructured data, and an event processing engine that is logicallyconnected between the at least one source of structured data and theevent module for generating events from the at least one source ofstructured data. The event module includes a user rule module includinga plurality of objects for use in allowing a user to define a userversion of at least one structured data or fact processing rule, and anevent table for storing generated events. The event processing engineincludes a receiving module for receiving data related to the at leastone source of structured data, a compiling module for obtaining theplurality of objects from the user rule module and generating aprocessing version of the at least one facts processing rule, and aprocessing module for evaluating the received data using the processingversion of the at least one fact processing rule and, in response to thereceived data matching the processing version of the at least one factprocessing rule, writing at least one event to the event table.

In one arrangement, the event module may include an alarming module forevaluating generated events in the event table using one or more alarmrules. In this arrangement, the alarming module may be operable togenerate a notification message for transmission to one or more entitiesin response to a generated event matching an alarm rule. In anotherarrangement, the event generating system may include at least onestructured data manager operatively interposed between the eventprocessing engine and the at least one data source. For instance, thestructured data manager may be operable to parse structured datareceived from the at least one data source, process the parsedstructured data, and forward matching structured data to the receivingmodule of the event processing engine for further analysis.

In one embodiment, the at least one fact processing rule may includefirst and second rule blocks (RBs), where the received data matches theprocessing version of the at least one fact processing rule upon a firstevaluation of the received data by the processing module using the firstRB being a first of at least first and second first RB outcomes and asecond evaluation of the received data by the processing module usingthe second RB being a first of at least first and second RB outcomes.For instance, the processing module may proceed to determine whether theother of the first and second evaluations is the other of the first RBfirst outcome and second RB first outcome in response to the first orsecond evaluations by the processing module comprising one of the firstRB first outcome and second RB first outcome.

In one arrangement, the processing module ceases determination ofwhether the other of the first and second evaluations is the other ofthe first RB first outcome and second RB first outcome upon expirationof a predetermined time period. For example, the predetermined timeperiod may be measured from a determination that the first or secondoutcome is the one of the first RB first outcome and second RB secondoutcome. In another arrangement, the first RB first outcome includes thereceived data matching one or more particular criteria and the second RBfirst outcome includes the received data comprising one or morequantitative fields having a value exceeding a particular thresholdvalue.

The structured or normalized data processed by the utilities disclosedherein may be generated in any appropriate manner and by numerous typesof devices. For instance, log messages and/or other data may begenerated by a variety of network platforms including, for instance,Windows servers, Linux servers, UNIX servers, routers, switches,firewalls, intrusion detection systems, databases, ERP applications, CRMapplications, and homegrown applications. The log data can be collectedusing standard network logging and messaging protocols, such as, forinstance, Syslog, SNMP, SMTP and other proprietary and non-proprietaryprotocols. Moreover, the log file may be text based, a proprietaryformat, a binary format, etc. In addition, the logs may be written todatabases such as Oracle, Sybase, MySQL, etc. As a result, a data systemmay generate a large number of logs in different formats, and it may bedesired to monitor or analyze these logs for a variety of purposes.Fields of information within such log messages can be identified and themessages can be selectively processed in accordance with rules based onthose fields. In this manner, enhanced processing of textual messagesincluding log messages may be achieved along with improved audit andcompliance analysis, application monitoring, security monitoring, andoperations analysis. Moreover, large networks may be supported andgrowing networks may be adapted to.

It should be appreciated that the various aspects discussed herein maybe implemented via any appropriate number and/or type of platforms,modules, processors, memory, etc., each of which may be embodied inhardware, software, firmware, middleware, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system that provides for management ofstructured data generated by one or more data platforms and eventsassociated therewith.

FIG. 2 is a block diagram illustrating a more detailed view of anadvanced intelligence engine (AIE) usable with the system of FIG. 1, andillustrates data flow and processing occurring among fact sources, theAIE, and an event manager.

FIG. 3 is a block diagram illustrating a more detailed view of one ofthe rule blocks (RBs) of an AIE rule shown in FIG. 2 according to oneembodiment, and illustrates data flow and processing occurring betweenone or more fact sources and the RB.

FIG. 4 is a block diagram illustrating a more detailed view of one ofthe rule blocks (RBs) of the AIE rule shown in FIG. 2 according to oneembodiment, and illustrates data flow and processing occurring betweenone or more fact sources and the RB.

FIG. 5 is a block diagram illustrating an evaluation schedule includingthree RBs.

FIG. 6 is a block diagram illustrating how a group of RBs cancollectively define two or more AIE rules, where the two or more AIErules may share a common RB.

FIG. 7 is a block diagram illustrating data flow and processingoccurring among a user console, an event manager, an AIE and a logmanager.

FIGS. 8-24 illustrate various screenshots of user interfaces that may bemanipulated by a user during the creation and modification of one ormore AIE rules made up of one or more rule blocks which may be utilizedby the AIEs, event managers, etc. disclosed herein to obtain usefulinformation from structured data such as processed logs.

FIG. 25 illustrates a data processing protocol for use with theteachings presented herein.

DETAILED DESCRIPTION

The present disclosure relates to computer network monitoring andinformation management through the processing of various types ofstructured or normalized data generated by or gleaned from devices,hosts or networks. The utilities disclosed herein are applicable to abroad variety of applications such as providing event detection forvirtually any type of system that generates data (e.g., computerservers, mainframes, network devices, security devices, access controldevices, etc.). While much of the present discussion will be in relationto log messages and other log-related data, it should be appreciatedthat the present utilities are applicable to numerous other types ofstructured or normalized data (e.g., forensic data, transactional data,activity data, and/or the like).

Turning to FIG. 1, a system 10 is provided that generally provides forthe collection, processing, management, and analysis of various types ofdata generated by or gleaned from one or more devices, networks,processes, and the like. As shown, the system 10 may includes one ormore root data sources 14 responsible for generating one or more typesof data 18 that may be analyzed in numerous manners to extractmeaningful information as will be discussed herein. The root datasources 14 may be represented by hosts or devices 22 (e.g., computers,servers, routers, switches) and networks 26 (although numerous otherforms of root data sources 14 are also envisioned), and may eachgenerate a plurality of text files describing various occurrences ordevelopments associated with the operations of the root data source 14.The generated text files may also be routinely updated by the root datasources 14 as various events transpire during the root data sources' 14operations, a process that may be referred to as “logging.”Additionally, while text files are often used for logging because oftheir readily manageable format and because a person can more easilyunderstand the information contained therein for diagnostic purposeswhen problems arise, data such as log messages may come in other formatsas well.

The root data sources 14 that generate the data 18 may come in a varietyof configurations, with each being capable of generating a tremendousamount of data 18 such as log messages. For example, one of the devices22 may be a computer (e.g., server, desktop, notebook, laptop, computerworkstation, mainframe system) that is operable within a computernetwork configuration. In this regard, the computer may be responsiblefor delivering applications to other devices 22 or processes within thecomputer network, administering communications among computers withinthe computer network, controlling various features of the computernetwork, and the like. In the process of performing these functions,although partially dependent upon the number of computers within thenetwork, the computer may generate thousands, millions, etc. of logentries per day. To illustrate, when a user incorrectly attempts tologon to a single computer on the computer network, the computer maygenerate a log entry noting a particular time (e.g., timestamp) that animproper procedure was performed. Other examples of occurrences ordevelopments that may cause the generation of log messages include,inter alia, application launch failures, audit activity, attacks,operating system errors, and the like.

While the data 18 may be in the form of log messages or entriesgenerated by or gleaned from root data sources 14, the data 18 may takenumerous other forms as well. For instance, the data 18 generated bydevices 22 may be in the form of host forensic data such as fileintegrity information, process information, data transfer information,and the like. As an additional example, the data 18 generated bynetworks 26 may be in the form of dataflows (i.e., recalculated valuesfor dependent variables that depend on one or more changing independentvariables), packet dumps, content inspection, and the like.

The system 10 of the present disclosure provides for the rapid/automatedextraction of viable information from the data 18. One component oraspect of the system 10 that facilitates this purpose is at least onelog or structured data manager 30 (e.g., a plurality of log managers 30)communicatively coupled (via any appropriate wired or wirelessnetwork(s)) to the various root data sources 14 to receive the data 18generated therefrom (e.g., collection). In this regard, the log manager30 may use various protocols (e.g., syslog protocols, Netflow protocols)to communicate with the root data sources 14. In one arrangement, thesystem 10 may employ agents or system monitors 34 (e.g., software) thatcan operate on the individual root data sources 14 to extract dataentries from registers or records of the root data sources 14. In someinstances, the system monitors 34 are software protocols that are innateto the operating system of a root data source 14.

The information that the log manager 30 may extract from the logs mayultimately be used to generate alarm messages that may be useful to anend user. For example, the log manager 30 may process thousands of logmessages and detect certain occurrences from the volume of datacontained therein by way of processing the received log messages againstone or more rules. The log manager 30 may aggregate log data into amanageable format that summarizes, for example, the frequency of aparticular event. Additionally, the log manager 30 may archive the abovedata for future reporting uses. This aggregation and archival maygenerally be referred to as management.

For instance, upon determining that a particular IP address attemptingto access one of the devices 22 or networks 26, the log manager 30 maygenerate at least one event that may be passed or transferred to anevent manager 38 to determine whether one or more alarms should begenerated (e.g., by processing the events against any appropriate alarmrule(s)). For instance, if the particular IP is that of a computer thatroutinely communicates to one of the networks 26 as part of anauthorized process, the event may simply be registered by the eventmanager for future use, if any. However, if the IP address belongs to acomputer system that is, for example, attempting to bombard the networkwith message traffic, the event manager 38 may generate an alarm that adenial of service attack is underway so that a system administrator maytake appropriate steps to limit or prevent any damage. Additionally, thetrends of events and/or alarm generation may be detected and reportspertaining to those trends may be provided. The various log managers 30and event managers 38 may transmit logs, events, alarms, alerts and/orother data to one or more third-party products 42 by way of anyappropriate third-party services 46. Representative examples of logmanagers 30, system monitors 34, event managers 38, and the like thatmay be used in conjunction with the system 10 may be found in U.S. Pat.No. 7,653,633 and U.S. Patent Application No. 61/360,815, the entiredisclosure of each being hereby incorporated herein by reference.

With continued reference to FIG. 1, the system 10 may include at leastone advanced intelligence engine (AIE) 50 that is broadly operable toanalyze and process numerous types of structured or normalized data(e.g., data 18 which may be received directly from the data sources 14;data which has been processed by one or more log managers 30; datarelated to identity, asset, configuration and vulnerability management,etc.) using one or more log processing, structured data, or AIE rules todetect what may be complex events/conditions/developments/etc. occurringin relation to the data sources 14 while not being limited to use oftraditional notions of “correlation”. Part of the analyses performed bythe AIE 50 may involve conducting one or more types of quantitative,correlative, behavioral and corroborative analyses to detect events fromone or more disparate data sources, even when the data would otherwisebe considered unimportant or non-relevant. Events generated by the AIE50 may be passed to the event manager 38 which to determine whetherfurther action is required such as reporting, remediation, and the like.

Turning to FIG. 2, a schematic block diagram of an AIE 100 (e.g., AIE 50of FIG. 1) is presented that illustrates one or more types of processingand data flows that occur as part of analyzing structured or normalizeddata collected from numerous sources (which may be disparate) togenerate events that may be further processed by an event manager 104(e.g., event manager 38 of FIG. 1). As will be described in more detaillater in this discussion, the AIE 100 (as well as the various othercomponents, modules, processes and the like) may be implemented by anyappropriate computer-readable code or logic stored on any appropriatecomputer-readable medium (e.g., RAM) and that is executed by anyappropriate processor(s) or controller(s).

The AIE 100 may include one or more AIE rules, where each AIE rule ismade up of one or more of what will be referred to as “rule blocks” (RB)107. Each RB 107 may include any combination of logic, processes, andthe like that is generally operable to obtain and store one or moretypes of possibly pre-filtered “facts” (e.g., structured data such aslog-related data), determine whether one or more “conditions” related tothe facts have been satisfied (e.g., the observation or non-observationof particular types of facts, exceeding or not exceeding a thresholdvolume of transferred data, and/or the like), and take one or more typesof actions upon satisfaction of the one or more conditions (e.g.,causing the evaluation of another RB 107 of an associated AIE rule,generating an event if the RB 107 is the only RB 107 of the AIE rule,generating an event if the conditions of all other RBs 107 of anassociated AIE rule have already been satisfied, etc.). The AIE 100 mayprocess numerous types of facts 124 generated by one or more platformsof one or more data systems, only some of which will be describedherein.

One of the types of facts 107 may be data received from one or more logor structured data managers 140 (e.g., log manager 30 in FIG. 1). Forexample, and as described herein and in U.S. Pat. No. 7,653,633 and U.S.Patent Application No. 61/360,815 which have been incorporated herein byreference, log managers may process log messages received from root datasources (e.g., root data sources 14 in FIG. 1) and other data againstany appropriate rule base made up of any appropriate expression syntaxand/or tagging notation. Upon a log message matching a rule, variousprocessing may be performed by the log manager such as determining logmessage direction (e.g., external, internal), calculating a risk-basedpriority, parsing meta-data from the log message text, and writing anyof the above along with the original log message text to a database ofthe log manager. Thus, various log managers may perform at least somefiltering of log messages and other data that is to be used by the AIE100.

Another type of fact 124 may be root data source data 144 (e.g., datastreamed directly from root data sources 18 in FIG. 1 or via one or morelog managers). Root data source data 144 may be at least substantialsimilar to the original data generated by the root data sources 18. Inone arrangement, each piece of root data source data 144 (e.g., each logmessage) may be a subset of parsed metadata contained in the logmessage. Another type of fact 124 may rule block result data 148. Ruleblock result data 148 may be any type of data generated by one or moreof the RBs 107 (e.g., metadata corresponding to satisfied conditions 128). For instance, a satisfied condition object 128 from one RB 107 may be“fed into” another RB 107 as a fact 124 for that RB 107. Thisarrangement may be useful when, for instance, one RB 107 is to betriggered for processing in response to a satisfied condition object 128generated by another RB 107. Numerous other types of facts 124 for usein the AIE 100 are envisioned such as time stamped occurrence of somekind, such as a LogRhythm Log record; a dynamic aggregate beingmaintained, such as recent Bytes sent for a user and host; staticreference information, such as the location and other attributes of aKnown Host, or the security permissions of a User; a timer-generatedFact to mark the start or end of a rule-based period; and the like. Itshould be appreciated that any appropriate protocol(s) may be usedduring communications between the AIE 100 and the various types of facts124.

When an AIE rule is made up of at least two RBs 107, the RBs 107 of eachadjacent pair of RBs 107 may be linked or connected by at least one“linking relationship” or “RB relationship” for reasons that will becomemore apparent in the ensuing discussion. With continued reference to therepresentative AIE 100 of FIG. 2, an AIE rule 120 includes RB₁ 108, RB₂112 and RB₃ 116, where RB₁ 108 and RB₂ 112 are linked by first linkingrelationship object 110, and where RB₂ 112 and RB₃ 116 are linked bysecond linking relationship object 114. Although AIE rule 120 has beenillustrated as having three RBs 107, the AIE rule 120 (and other AIErules) may, in other embodiments, have more or less than three RBs 107.

Each linking relationship object generally defines which fieldcontent(s) of processed facts in one RB 107 is(are) to be used as a keyor keys into the processed facts in an adjacent RB 107 to determine ifthe adjacent RB's 107 condition has been satisfied as to the key(s).Each linking relationship object may also define a particular timeperiod (as measured from any appropriate reference time, such as thesatisfaction of a condition of an adjacent RB 107) within which acondition of an adjacent RB 107 must be satisfied in order for an eventto be generated. More specifically, upon one RB 107 firing (i.e., uponthe condition of the RB 107 being satisfied), the AIE 100 then proceedsto evaluate whether the conditions of one or more adjacent RBs 107 havebeen satisfied in relation to the particular equated field content(s) inthe linking relationship object(s).

For instance, assume the “classification” fields of first and secondadjacent RBs 107 were equated in a linking relationship object and thatone of the first and second adjacent RBs 107 fired. In this case, thespecific classification field content (e.g., “attack,” “compromise,”“suspicious,” or the like) of the satisfied condition of the one RB 107that fired would be used as a key into an index structure of data of theother RB 107 (discussed in more detail below). That is, the AIE 100would evaluate the other RB 107 to determine if its condition has beensatisfied in relation to the particular specific classification of thefirst RB 107. As another example, an “impacted” unit (e.g., host,network, region, etc.) in one RB 107 could be equated to a corresponding“origin” unit (e.g., host, network, region, etc.) in an adjacent RB 107.The first and second linking relationship objects 110, 114 may be thesame or different. Numerous other examples of linking relationshipobjects are envisioned (e.g., equating one or more of direction, domain,entity, etc. fields of adjacent RBs). As will be discussed in moredetail below, the fields specified in linking relationship objects maybe “group by” fields (i.e., fields by which log messages processed by anRB 107 will be aggregated for use in AIE rule execution). A similarprocess would occur with other RBs 107 of the particular AIE rule inquestion.

In any case, and upon it being determined that each of RB₁ 108, RB₂ 112and RB₃ 116 has a satisfied condition 128, one or more events 133 may begenerated in any appropriate manner and eventually passed to the eventmanager 104 for further processing (e.g., such as to determine whetheralarms, alerts or remediation should be sent or performed). In onearrangement, the final RB that determines its condition has beensatisfied may function to generate the event 133. In anotherarrangement, a separate module or manager (e.g., an “event generator”)may be provided that operates to receive satisfied condition objects 128from RBs, confirm that all RBs of a particular AIE rule have generatedsatisfied condition objects (or otherwise that their conditions havebeen satisfied), and correspondingly generate one or more events 133. Inany case, each event 133 may be in the form of an object or messageincluding any appropriate number of metadata fields that summarize theevent (e.g., source IP address, bytes of data transferred out of thenetwork, etc.).

The one or more RBs 107 can allow what may be complex events to bedetected from disparate types of data over various time periods orincrements. For instance, a network administrator may be interested inknowing about any server backup processes that started but neverfinished. Here, one RB 107 (e.g., RB_(A)) could be designed to fire whena backup for any internal server has started, and a second RB 107 (e.g.,RB_(B)) could be designed to fire when no “server backup completion”messages have been received or generated within a particular period oftime after the server backup process begins (e.g., 5 hours). A linkingrelationship object between RB_(A) and RB_(B) may specify, for instance,that A.IPaddress(destination)=B.IPaddress(origin). In this regard, uponthe condition of RB_(A) being satisfied (e.g., RB_(A) detecting that aninternal server backup has started, which may be indicated in a logmessage possibly generated by the internal server), the AIE 100 wouldthen obtain the specific IP address in the destination IP address fieldassociated with the server backup process and use it as a key into anindex structure of the data of RB_(B) to determine whether the specificIP address exists in an origin IP address field of a server backupcompletion message in the 5 hours after the server backup processstarted.

With additional reference now to FIG. 3, a schematic block diagramillustrating processing performed at or by an RB 107 (e.g., one or moreof RB₁ 108, RB₂ 112, or RB₃ 116) is shown. The RB 107 may generallyinclude one or more filters that are operable to perform a conditionalcheck on some part of the incoming facts 124 to either subject thefact(s) 124 to further processing (e.g., as part of determining whethera “condition” of the RB 107 has been satisfied) or reject or otherwiseignore the fact(s) 124. As shown, the RB 107 may include a filteringmodule 152 that may include a “primary criteria” filter 160 where allspecified filters must be matched in order for a log message to becollected and processed. For instance, a user may choose to filterincoming facts 124 based on a particular classification (e.g., “attack,”“compromise,” “suspicious,” or the like) of the facts 124 (where theclassification of a fact 124 may be contained within a particularmetadata field associated with the fact 124). As additional examples, auser could additionally or alternatively choose to filter incoming facts124 based on a particular originating country (e.g., China, Syria,etc.), impacted IP address, etc. of the fact 124.

Another filter may be an “include” filter 164 where if one or moreinclude filters 164 are specified, an incoming fact 124 must match atleast one of the specified include filters 164 in order for the fact 124to be collected and processed. Another filter may be an “exclude” filter168 where if one or more exclude filters 168 are specified, an incomingfact 124 cannot match any of the exclude filters 168 in order for thatfact 124 to be collected and processed. Another filter may be a day/timefilter 172 where a fact 124 must fall within any specified day of weekand/or time of day filters (as determined by any appropriate time stampassociated with the fact 124). It should be understood that theaforementioned filters are only for exemplary purposes and the presentdisclosure should not be construed as being limited to such filters.

For instance, one or more of the filters in the filtering module 152 maybe used to filter for a particular type or types of log messages as partof what may be referred to as a “log” RB 107. That is, the filteringmodule 152 may be used to monitor for or observe a particular type oflog message or messages. For instance, the filtering module 152 may beused to detect that a known malicious IP address has made contact with aparticular network device (e.g., device 22 in FIG. 1) by way ofreceiving and filtering for log messages generated by the device 22indicative of the contact by the malicious IP address.

In any case, the initial filtering module 152 may store filtered facts124 (i.e., an outcome of the evaluation) and/or metadata (e.g., IPaddresses, classifications, TCP/UDP ports, etc.) related to the filteredfacts 124 in at least one “in-memory” fact or data structure 156 (e.g.,storage, container, table, etc.) in any appropriate manner. Furthermore,at least one indexing structure 158 may also be provided to allow forrapid lookups and quick access of facts 124 and/or related metadata inthe data structure(s) 156. Each indexing structure 158 may be organizedor constructed according to the particular linking relationship object182 that links the RB 107 to one or more other RBs 107. For instance, ifa linking relationship object 182 specified that an impacted IP addressin a first RB 107 was to be equated to an origin IP address in a secondRB 107, then the indexing structures 158 of each of the first and secondRBs 107 could include a list of IP addresses, each of which identifiesand allows for access to a corresponding filtered fact(s) 124 in thedata structures 156. Upon one of the first and second RBs 107determining that its condition has been satisfied and passing thespecific IP address of the fact(s) 124 that caused the condition to besatisfied to the other of the first and second RBs 107 (e.g., as part ofa satisfied condition object 128 ), the other of the first and secondRBs 107 could then utilize the specific IP address as a key into itscorresponding indexing structure 158 to determine whether its conditionhas been satisfied as to the specific IP address. This arrangementadvantageously facilitates the rapid determination as to whether RBconditions have been satisfied and as to whether events of an AIE ruleencompassed by one or more RBs should be generated.

In one arrangement, the AIE 100 may automatically determine the type andformat of one or more indexing structures 158 to enable the efficientevaluation of AIE rules. For example, assume a first RB 107 (RB₁)filters for facts according to IP address and host name, and a second RB107 (RB₂) filters for facts according to host name and user. Also assumethe linking relationship object between RB₁ and RB₂ specifies thatRB₁.Host=RB₂.Host. Here, the AIE 100 (e.g., via a translator) mayautomatically determine that RB₁ needs a primary index structure 158 onIP:Host and a secondary index structure 158 on host due to its symmetriclink with RB₂ on Host (e.g., as RB₂ could fire, requiring RB₁ toevaluate its condition using the key of the specific content of the hostfield received from RB₂). Likewise, the AIE 100 may determine that RB₂needs a primary index structure 158 on Host:User and a secondary indexstructure 158 on Host. In the case where an additional RB₁ (e.g., athird RB 107 such as RB₃) is to use the data from RB₁ (e.g., either aspart of a current AIE rule or a different AIE rule) where a linkrelationship object between RB₁ and RB₃ involves an IP address, the AIE100 would automatically add an additional secondary index to RB₁ on IP(which would not have been necessary in the former example).

With continued reference to FIG. 3, the RB 107 may also include acondition manager 176 that operates to determine whether a satisfiedcondition object 128 is to be generated based on the presence and/orabsence of particular filtered facts 124 in the data structure 156. Thecondition manager 176 may make such determinations by way of accessingthe indexing structure 158 and/or data structure 156 and may do soaccording to any appropriate schedule (e.g., seconds, minutes, hours),upon newly filtered facts 124 being received and/or stored, and thelike.

For instance, a user may specify that the condition manager 176 is togenerate a satisfied condition object 128 upon the initial filteringmodule 152 “observing” or otherwise collecting and storing one or moreparticular types of facts 124 in the data structure 156 (e.g.,collecting a fact 124 that includes the above discussed malicious IPaddress, collecting facts 124 that have a classification of“compromise,” etc.). As another example, a user may specify that thecondition manager 176 is to generate a satisfied condition object 128upon the filtering module 152 “not observing” and thus not storing oneor more particular types of facts 124 (e.g., during a particular timeperiod). For instance, the filtering module 152 could be designed todetect whether a particular router or server has not generated a“heartbeat” type log message during every particular time period (e.g.,every 30 minutes). In this example, the condition manager 176 couldgenerate a satisfied condition object 128 upon the RB 107 failing toreceive a heartbeat message from the particular router or server at theend of a 30 minute segment. A user could also specify that the conditionmanager 176 is to generate a satisfied condition object 128 upon thefiltering module 152 “not observing” one or more particular types offacts 124 after another (i.e., adjacent) RB 107 has generated asatisfied condition object 128.

As shown, the condition manager 176 may also include or have access to alinking relationship object 182 which generally represents a linkingrelationship between this RB 107 and one or more adjacent RBs 107 (e.g.,in relation to specific fields between adjacent RBs 107 that are to beequated, particular time periods within which respective conditions ofthe adjacent RBs 107 must have been satisfied in order for an event topossibly be generated, etc.). In the case where the condition manager176 of this RB 107 has determined that its condition has been satisfied(e.g., it has “observed” particular facts), the condition manager 176may access the linking relationship object 182 to determine which of thefields of the observed facts is to be extracted and passed to anadjacent RB 107 (e.g., for the adjacent RB's 107 determination as towhether its condition has been satisfied in relation to the extractedfield). In the case where the condition manager 176 has received alinking relationship field content (e.g., a linking “key”) from anadjacent RB 107 (the specific field content, e.g., IP address, extractedas part of an adjacent RB 107 determining that its condition has beensatisfied), the condition manager 176 may then utilize the linking keyto determine if a corresponding entry exists in its correspondingindexing structure 158, and then make a decision as to whether itscondition has been satisfied based on a result of the determination.

With continued reference to FIG. 3, a generated satisfied conditionobject 128 may include metadata 184 such as filtering results 188 (e.g.,time range of the query, quantitative values that led to a successfultriggering of the RB 107, time stamps, the totals of filtered logmessages according to selected “group by” parameters, and/or the like),linking keys 187, and the like. In the case where an RB 107 is the firstRB 107 of an AIE rule to have its condition satisfied, the satisfiedcondition object 128 may be passed to adjacent RBs 107 which may utilizethe metadata 184 as part of determining whether their respectiveconditions have been satisfied and thus whether events are to begenerated. In the situation where an RB 107 is the last RB 107 of an AIErule to have its condition satisfied, the RB 107 may either function tocause the generation of an event 133 by itself or else pass a satisfiedcondition object 128 to an event generator which may function to do thesame.

In any case, and upon the satisfaction of all RBs 107 of an AIE rule andthus satisfaction of the AIE rule as a whole, the metadata 184 of theone or more RBs 107 making up the AIE rule may be written (e.g., in astructured format) to one or more fields of an entry in an event datastore (where the various written data collectively makes up an “event”).In addition to the various pieces of metadata 184, other data may bewritten such as a value that identifies the specific AIE rule used togenerate the event, the timestamp of the last edit time of the AIE rule,and the like. The writing of events to an event data store will bediscussed in more detail below.

With reference to FIG. 4, another embodiment of an RB 107′ is shown.Corresponding components/modules between the embodiments of FIGS. 3 and4 are identified by common reference numerals and those that differ inat least some respect are identified by a “single prime” designation inFIG. 4. For clarity, the various components/modules of the filteringmodule 152, condition manager 176 and satisfied condition object 128′have not been shown. One difference between the RB 107 of FIG. 3 and theRB 107′ of FIG. 4 is the inclusion a “thresholds” object 194 and/or a“unique values” object 196 that the condition manager 176′ of the RB107′ uses to determine whether a satisfied condition object 128′ is tobe generated. More specifically, the thresholds object 194 allows a userto specify one or more particular thresholds for quantitative fields ofdata in the data structure 156 (as filtered by filtering module 152)which the condition manager 176′ uses as part of determining whether ornot to generate a satisfied condition object 128′. Examples of fieldsthat may be considered for threshold consideration include, but are notlimited to, log count, bytes in, bytes out, total bytes, items in, itemsout, total items, quantity, amount, rate, size, duration, and/or thelike. For instance, a user could use the thresholds object 194 tospecify a log count of 1,000. The user could also specify that thecondition manager 176′ is to generate a satisfied condition object 128′when a log count of 1,000 has been observed within any appropriateperiod of time (e.g., within 24 hours).

For instance, the condition manager 176′ may query the data structure156 according to any appropriate schedule and/or when new facts arrivein the data structure 156 to determine whether a log count of 1000 hasbeen reached in a 24 hour period. It should be appreciated that acondition that was false just minutes ago (e.g., log count of 900 in 24hour period) could suddenly become true (e.g., log count of 1500 in 24hour period), such as with a sudden influx of facts 124, or simply byvirtue of the 24 hour window continually shifting (e.g., the time windowmay not encompass a log count of 1000 one minute and encompass a logcount of 1000 the next minute). In this regard, the condition manager176′ may be configured to query the data structure 156 for thresholds asoften as appropriate based on the particular type of threshold beingmonitored. In one arrangement, the condition manager 176′ may query thedata structure 156 according to predetermined schedule (e.g., every 5minutes) when the particular count or quantity is relative low (e.g.,less than 70% of the threshold), and then dynamically begin querying thedata structure 156 upon each new fact 124 or group of facts 124 beingstored in the data structure 156, such as in relation to somewhatpredictable counts (e.g., to conserve processing resources).

Alternatively, the user could specify that the condition manager 176′ isto generate a satisfied condition object 128′ upon a particularthreshold not being observed within a particular time period (e.g., alog count of 1,000 not being observed within any 24 hour period, or notbeing observed with 24 hours after an adjacent RB's 107 condition hasbeen satisfied. Users may also use the thresholds object 194 to specifythresholds for more than one field, and may specify that satisfaction ofone or all of the thresholds is required for satisfied condition object128′ generation. Furthermore, the filtering results 188 in the metadata184 of the pending event 128′ (see FIG. 3) may include various types ofinformation such as the specific quantitative field(s) and threshold(s)observed or not observed, the specific value of the particularquantitative field reached upon or before generation of the pendingevent 128′, the time limit or period within which the threshold wasobserved or not observed, time stamps, and/or the like.

With continued reference to FIG. 4, the “unique values” object 196allows a user to specify one or more unique values or occurrences thatmust be observed or not observed in the facts 124 in the data structure156 as processed by the filtering module 152 in order for a satisfiedcondition object 128′ to be generated. Fields such as log source entity,log source host, direction, domain, sender, and the like may be used forunique value consideration. As an example, an administrator may use theunique values object 196 to specify a condition for this RB 107′ such as“more than three unique destination IP addresses for a given source IPaddress observed” (or not-observed) in the data structure 156.Furthermore, a user may specify a time period within which theparticular count of unique values must be observed or not observed forgeneration of the satisfied condition object 128′. The filtering results188 in the metadata 184 of the satisfied condition object 128′ mayinclude various types of information such as the specific uniquevalue(s) observed or not observed, the count of the unique value(s)observed or not observed, the time limit or period within which aparticular count of a unique value(s) was or were observed or notobserved, the particular fields being considered for unique values atthe time of satisfied condition object 128′ generation, and/or the like.

In one arrangement, the AIE 100 may maintain a “unique value” indexingstructure 158 that allows for the efficient determination of the countof unique values seen of a specific data field (e.g., destination IPaddresses) of facts over a particular interval of time. Initially, theAIE 100 may monitor for the number of unique values of the specific datafield (e.g., the key) in each of a plurality of time intervals (e.g.,“bins”), where the sum of the various bins equals the particular timeperiod over which the unique values are to be measured. For instance,for a 30 minute time period, the AIE 100 may monitor three 10-minutebins, thirty 1-minute bins, etc. In any case, the AIE 100 may thenperform a query to examine the bins and efficiently ascertain the totalnumber of unique occurrences of the key in the 30 minute time period(e.g., via performing “set union” operations with like-type sets fromother bins). The AIE 100 may automatically request creation of a uniquevalues indexing structure 158 to efficiently manage RB queries.

Turning now to FIG. 5, a schematic block diagram is shown illustrating arepresentative timeline 200 of RB evaluation or consideration for an AIErule 216 made of RB₁ 204, RB₂ 208 and RB₃ 212. As discussed previouslyand as will be discussed in more detail later in this discussion, a usermay specify (e.g., via any appropriate user interface in communicationwith the AIE 50 or 100) any appropriate temporal manner in which RBs areto perform evaluations to determine whether their conditions have beensatisfied. In some situations, a user may be able to specify that an RBis to perform an evaluation in relation to the time of the day andirrespective of when other RBs are evaluating their conditions or whentheir conditions are (if at all) satisfied. More specifically, the usermay specify a time zone, start time, end time, frequency, one or moremonitoring intervals, and/or the like for evaluation of an RB. Forexample, if a user specified a start time of 1:00 AM and a frequency of5 minutes, evaluations could occur between 1:00:00 and 1:04:59, 1:05:00and 1:09:59, 1:10:00 and 1:14:59, etc. until an end day or time wasreached (which may cease before any specified end time if the RB hasgenerated a satisfied condition object 128 or 128′).

In other situations, additionally or alternatively, a user may be ableto specify that an RB performs its evaluation in relation to when anadjacent RB's condition was satisfied. For instance, a user may be ableto specify that upon an RB receiving a satisfied condition object orother notification that an adjacent RB's condition was satisfied, the RBmay begin its evaluation for any desired period of time. Furthermore,such desired period of time may be effectively offset in relation to thetime at which the adjacent RB's condition was satisfied (e.g., asmeasured from the timestamp of the filtered fact(s) 124 that lead to thedetermination that the adjacent RB's condition was satisfied as opposedto the time that the adjacent RB determined that its condition wassatisfied or the time that the current RB received notification that theadjacent RB's condition was satisfied).

In one embodiment, one RB may begin its evaluation for a period of timethat starts upon satisfaction of an adjacent RB's condition (e.g., wherethe adjacent RB detects that an external IP address connected with aninternal IP address, and the one RB is detecting for large datatransfers from an internal IP address to an external IP address in the30 minutes immediately after the external/internal connection). Inanother embodiment, one RB may be configured to effectively begin itsevaluation at a time before the adjacent RB's condition was satisfied.More specifically, upon an adjacent RB's condition being satisfied at aparticular time (e.g., time of the timestamp(s) of the filtered fact(s)124 leading to the determination that the adjacent RB's condition wassatisfied), the one RB may evaluate its filtered facts 124 beginningwith filtered facts 124 associated with timestamps that are some periodof time earlier than the time at which the condition of the adjacentRB's was satisfied.

It is also noted that in many situations, any of the RBs of a particularAIE rule may determine that its condition has been satisfied first(i.e., regardless of an order in which the RBs are configured), wherebyadjacent RBs may then make such determinations. More specifically, whileit may be logical conclusion that a first RB would detect anexternal/internal connection before a second RB would detect large datatransfers from the internal device in the 30 minutes after theexternal/internal connection, processing delays may cause the second RBto detect large data transfers which occurred after an external/internalconnection before the first RB detects the connection. In this regard,upon the second RB detecting such a large data transfer, the first RBwould then be operable to determine that its condition was satisfied ifit detected an external/internal connection in the 30 minutes before thetime at which the second RB determined that its condition was satisfied(i.e., measured by the time stamp of the filtered fact(s) 124 associatedwith the large data transfer. The various evaluation time periods forRBs as measured from adjacent RBs may be encapsulated within linkingrelationship objects that link adjacent RBs (e.g., linking relationshipobjects 110, 114 in FIG. 2 and 182 in FIGS. 3-4).

In the representative timeline 200 of FIG. 5, RB₁ 204 begins itsevaluation at time t₁ 220, and a satisfied condition object 128 or 128′is generated at time t₂ 224 (although it should be understood that RB₁204 and other RBs do not necessarily generate a satisfied conditionobject 128 or 128′ upon every evaluation; the illustration in FIG. 5presumes that every evaluation of a RB results in generation of asatisfied condition object 128 or 128′ merely for facilitatingunderstanding of the representative timeline 200). In this example, auser has specified that RB₂ 208 is to begin its evaluation at time t₃232 which is offset from time t₂ 224 (i.e., the timestamp of thefiltered fact(s) 124 which caused RB₁ 204 to generate a satisfiedcondition object 128 or 128′) by time interval 228. Also, RB₃ 212 beginsevaluation at time t₄ 236 which is the same time as (or shortly after)RB₂ 208 generates a satisfied condition object 128 or 128′.Alternatively, RB₃ 212 may begin evaluation based in any appropriaterelation to time t₁ 220, t₂ 224, t₃ 232 or t₄ 236 (e.g., offset from oneof t₁ 220, t₂ 224, t₃ 232 or t₄ 236). Upon each of RB₁ 204, RB₂ 208 andRB₃ 212 generating satisfied condition objects (or otherwise determiningthat their conditions have been satisfied), an event may be generated.

While evaluation by one or more of the RBs has been discussed inrelation to time, it should be appreciated that such evaluation maycommence in response or relation other parameters or occurrences aswell. For instance, receipt of new or updated filtered facts 124 maytrigger (in any appropriate manner) one or more of the RBs to perform anevaluation (or a reevaluation as appropriate) of the updated facts 124(which may be together with the existing facts 124). Turning now to FIG.6, a collection 300 of RBs is provided to illustrate how first andsecond AIE rules 301, 302 (or additional AIE rules) may share at leastone common RB from the collection 300 of RBs in various manners. Forinstance, the first AIE rule 301 may include RB₁ 304, RB₂ 308 and RB₃312 while the second AIE rule 302 may include RB₁ 304, RB₄ 316 and RB₅320. This arrangement advantageously conserves or otherwise makes betteruse of system resources by sharing at least one common RB (e.g., RB₁304) between multiple AIE rules. For instance, upon RB₁ 304 generating asatisfied condition object 128 or 128′, each of RB₂ 308 and RB₄ 316could be evaluated, which evaluations, as discussed previously, maycommence in any desired relation to evaluation commencement or satisfiedcondition object 128 or 128′ generation of RB₁ 304. Evaluations by RB₃312 and RB₅ 320 may also occur according to any desired scheduling. Itshould be appreciated that administrators may construct various complexarrangements of RBs to define any appropriate number of AIE rules.

With reference to FIG. 7, a system 400 is shown that illustratescomponents/modules of and interrelationships among a source of log datasuch as a log or structured data manager 412 (e.g., log manager 30), auser console 416, an event manager 408 (e.g., event manager 38, 104)logically connected and/or interposed between the log manager 412 andthe user console 416, and an AIE 404 (e.g., AIE 50, 100) that islogically connected and/or interposed between the log manager 412 andthe event manager 408. As will be appreciated from the ensuingdiscussion, AIE rules (e.g., AIE rules 120 in FIG. 2) may generally haveat least two distinct instances: an AIE “user” rule 440 which is theinstance that a user/administrator interacts with to configure thefunction of an AIE rule (e.g., via user console 416) and, statedotherwise, exposes the properties and settings of a AIE rule formanagement by the user (e.g., RBs for promoted event generation, alarmrules associated with the AIE rule that determine when an alarm is to begenerated, etc.); and an AIE “engine” rule 452 which is the result ofconverting an AIE user rule into objects and/or structures that enablethe AIE 404 to detect a desired event (i.e., the AIE engine rule is acompiled, run-time version of the AIE user rule 440 within the AIE 404).

A user may utilize the user console 416 to create/edit/delete AIE rulesand view events, alarms and other messages generated as a result of AIEprocessing. The user console 416 may be in the form of any appropriatesoftware package that may be run or otherwise manipulated on anyappropriate computing device or system having memory for storing logic,a processor for executing the log, a display for viewing the results ofprocessor execution, and the like. The user console 416 may be inappropriate communication with the log manager(s) 412, event manager(s)408, AIE 404, and the like to both allow a user to manipulatecomponents, modules and managers of the system 400 as well as receivevarious types of information in relation to logs, events, alarms, andthe like.

For instance, the user console 416 may include an AIE rule manager 420that exposes AIE user rules 440 for management, importing and exporting,and a message engine 424 that displays events and any associated alarmsgenerated as a result of the events while allowing for searching andother types of manipulation of the events, alarms, etc. In onearrangement, the message engine 424 may be accessible on a “personaldashboard” of the user console 404 and may include the ability toinvestigate events/alarms/etc. generated as a result of AIE 404processing (e.g., searching, one-click correlation, etc.), generatereports, and the like.

In any case, AIE user rules 440 created or edited by users via the AIErule manager 420 may be passed to and stored on a data store 428 of theevent manager 408 as a number of separate serialized objects that may beused for various purposes. For instance, during creation of an AIE userrule 440 (which, as discussed previously, includes one or more RBs, suchas RBs 107 of FIG. 2), a user may configure a number of AIE user rule440 properties, such as one or more of the following parameters orproperties:

Property Description AIE User Rule ID A unique ID for the AIE User Ruleas stored in the SysParm data store 428. Name The Name of the AIE UserRule. Brief Description A brief description of the AIE User Rule.Details Additional details pertaining to the AIE User Rule. Group Agroup the AIE User Rule is associated with that is used to organize therules by type/function. AlarmRuleID An underlying Alarm object that isdirectly associated to the rule and may handle notification and/oreventual automatic remediation. CommonEventID A unique Common Event thatis directly and solely tied to the AIE User Rule. This is the CommonEvent saved as part of the Event record and may carry with itClassification and Risk Rating. RBs Collection A collection containingone or more ordered RBs RB Relationships Describing how each RB isrelated to another RB. Suppression The amount of time in seconds thatidentical events should be suppressed following an immediately precedingEvent. IsEnabled Enabled|Disabled. RecordStatus Active|Retired.ExpirationDate The datetime in UTC that the AIE User Rule should beconsidered disabled. Permissions Permissions for the AIE User Rule.PersonID The owner of the rule (if applicable). DateUpdated The lastdate record updated. Version The version of the AIE User Rule which maybe used for forward/backward compatibility. RecordType Indicates if theAIE User Rule is a System or Custom rule.

In one arrangement, the data store 428 may save the various propertiesas at least two serialized objects. One of the objects may be an AIErule properties object 444 that includes various properties that may beneeded by the AIE 404 for fact processing (e.g., facts 124 in FIG. 2),pending and promoted event generation, and the like (e.g., AIE User RuleID, CommonEventID, Suppression, RB Relationships, etc.). Another of theobjects may be an alarm rule object 448 which contains properties thatmay be needed by an alarm rule manager (ARM) module or service 436 ofthe event manager 408 for use in determining whether alarms, alerts,notifications, and the like should be generated upon the generation ofan event by the AIE 404 (e.g., AlarmRuleID). For instance, an alarm rulemay be designed to match on the CommonEventID associated with the AIErule. The AIE 404 and ARM service 436 may appropriately query the datastore 428 for the above discussed objects for use in log/event/alarmprocessing. In this regard, there may be an alarm rule created for everyAIE rule that is then used to notify personnel when the AIE rule issatisfied and a corresponding event is generated.

The AIE 404 is, as discussed throughout this disclosure, generallyresponsible for obtaining and performing processing on facts based onone or more AIE rules and determining whether events should be generatedfor use by personnel in making remediation decisions, for furtherprocessing by an event manager, and the like. For instance, the AIE 404may include a receiving module or communication manager 456 thatcoordinates retrieval of facts 124 log manager 412) for processing bythe RBs of the AIE rules.

Before discussing additional aspects of the AIE 404, it may be useful toidentify features of the log manager 412 (or log managers) that mayfacilitate access to log manager data by the AIE 404. As discussed inthe aforementioned patent and patent application that have beenincorporated herein by reference, the log manager 412 may include aprocessing engine and a mediator server which are collectivelyrepresented by reference numeral 460. The processing engine may begenerally responsible for parsing received log messages, processing thereceived log messages (e.g., against one or more processing rules),assisting with event preparation, and the like. The mediator server maygenerally be responsible for handling connections from log agents orsystem monitors (e.g., system monitors 34 in FIG. 1), insertingforwarded data into databases, handling the archiving and/or destructionof log data, and the like.

The mediator server may also be configured (via an AIE configurationfile 464) to forward any desired type and amount of log messages (e.g.,identified log metadata) to the communication manager 456 by way of alog forwarding engine 468. For instance, upon the mediator serverlocating the AIE configuration file 464 and confirming that the file isvalid, the mediator server may begin to forward (e.g., via the logforwarding engine 468) one or more log messages to the communicationmanager 456 of the AIE 404. The mediator server may pass log messages tothe communication manager 456 as soon as they are received at themediator from one or more root data sources (i.e., the mediator servermay “stream” the data to the communication manager 456), according toany desired schedule and/or in response to any desired occurrence.

The log manager 412 may also have a system monitor 472 that is operableto monitor the log forwarding performed by the log forwarding engine 468(e.g., by processing log messages generated by the log forwarding engine468 related to the forwarding against any appropriate log processingrules). This feature may allow administrators to diagnose and remediateproblems in relation to log forwarding from the log manager 412.

With reference again to the AIE 404, a compiling module 450 may receiveAIE rule property objects 444 from the from the data store 428 alongwith rule data structures 445 and compile the AIE rule property objects444 into AIE engine rules 452 that are prepared for execution orprocessing by a processing module 476. The processing module 476 mayprocess any appropriate facts received from the communication manager456 utilizing the AIE engine rules 452 to, as discussed previously,generate events. The processing module 476 may write events into anyappropriate database, table or other data structure in an event datastore 432. For instance, a written event may include the CommonEventIDassociated with the particular AIE User Rule ID and/or any uniquemetadata values that can be populated based on AIE rule satisfaction.

As discussed previously, metadata from RBs and/or other information(e.g., value(s) that identifies the specific AIE rule used to generatethe event, the timestamp of the last edit time of the AIE rule, and thelike) making up an event may be written to one or more fields of anevent entry in an event data store. The written data, along with theoriginal definition of the AIE rule, may enable subsequent analyticprocesses to take a specific event and perform a query against one ormore log or structured data managers (or other data sources) to obtain,for each RB of the AIE rule, a set of data that meets the criteria ofthe AIE rule. As AIE rule definitions may be changed, a current editdate of the AIE rule may be compared with the version saved in the datamaking up the event to help provide a user with a “confidence value”that the result set will accurately reflect the definition of the rule.For instance, FIG. 24 presents a screenshot of a user interface 1100that illustrates XML, data 1104 that processing module 476 of the AIE404 recorded or wrote for a particular event. As seen, the XML data maybe compactly recorded and may include the specific criteria and timeperiod involved for each of a number of RBs, thus enabling drilldownqueries and the like to be run based on the XML data 1104.

Advantageously, an event can be written as soon as its associated AIErule is minimally satisfied, even if additional data coming in makes therule “more true” (e.g., data that continues to fall under the compoundquery conditions for a single firing of the AIE rule.). For example, anAIE rule monitoring for a threshold of data sent over a 10 minute periodcould conceivably be triggered by the arrival of a single fact (e.g., asingle log message) with a quantity that exceeded that value, even ifmany additional facts continue to arrive during a 10-minute windowmaking the total even greater). Additionally, numerous possiblyexpensive database updates can be avoided or at least limited bylimiting the writing of the same event multiple times

Furthermore, users may view (e.g., in a single screen) the data for eachRB that falls logically under the definition of an AIE rule. Forexample, a complex 3-RB AIE may list data that satisfied RB1, then RB2,then RB3. A user may then freely view all the additional fields in thosefacts (e.g., log records) that did not make up the core AIE ruledefinition (e.g., to help determine what may have been going on). Forinstance, an AIE rule may only have specified conditions related to IPaddresses, network Zones, and data quantities transmitted; however, auser may be able to view additional fields and ascertain that the sameuser is involved in many of those logs, thus raising suspicion.

Like the log manager 412, the AIE 404 may have a system monitor 480 thatis operable to monitor the functionality of the variouscomponents/modules of the AIE (e.g., the communication manager 456, thecompiling module 450, the processing module 476, etc.) by way ofprocessing log messages generated by such components/modules against anyappropriate log processing rules. Furthermore, the AIE 404 may also beconfigured by way of any appropriate configuration file 484. The ARMservice 436 may retrieve any appropriate alarm rule objects from thedata store 428 and events from the event data store 432 and performevaluating or processing to determine whether alarms, notificationsand/or the like should be generated. It should be appreciated that anyappropriate protocols may be utilized for governing communication amongand between the various components/modules of the system 400.Furthermore, the various components/modules may be implemented withinseparate servers, a common server, and/or the like.

Turning now to FIG. 8, one screenshot of a user interface 500 is shownthat may be presented on a screen of a user's computing device inconjunction with the creation/editing of AIE rules performed by the AIErule manager 420 of FIG. 7. As discussed above, the AIE rule manager 420may be made accessible to a user via the user console 416. The firstpart of the ensuing discussion will be in relation to a number ofscreenshots of a user interface that allow a user to configure a logprocessing or AIE rule as a whole. The second part of the ensuingdiscussion will be in relation to a number of screenshots of a userinterface that allow a user to configure specific RBs that make up aparticular AIE rule. The third part of the following discussion will bein relation to a number of screenshots of a user interface that allow auser to observe events and drill down into one or more specific eventsto obtain more detailed information related to the events. It should beunderstood that the AIE discussed herein is not be limited to use withthe particular screenshots and user interfaces shown in the figures.Rather, these figures have only been provided to assist the reader inunderstanding how an AIE rule may be created or manipulated on a userconsole, according to one embodiment.

As seen in FIG. 8, the user interface 500 may include a number of tabs508 that allow a user to toggle between various screens of the userinterface 500. One of the tabs may be a “Rule Blocks” (RB) tab 508 thatallows a user to select the particular types of RBs that are to be usedin a particular AIE rule. Upon selection of the RB tab 508 (asillustrated in FIG. 8), the user interface 500 may display a number ofregions for use in selecting particular RBs, defining theirrelationships, setting an evaluation offset schedule, etc. For instance,one region 512 may display a number of buttons 516 representative of thevarious RB types that may be included as part of an AIE rule (e.g., log,threshold, unique values). Manipulation (e.g., clicking) of one of thebuttons 516 displays various graphical icons 520 that represent thevarious manners in which a particular RB would cause the generation of asatisfied condition object. As an example, and as shown in FIG. 8, upona user selecting the “log” button 516, and clicking and dragging the“observed” graphical icon 520 into the “rule block designer” region 524,an RB would be added to the particular AIE rule, where the RB wouldgenerate a satisfied condition object upon particular a particular typeof log message being observed by the RB (e.g., such as log dataclassified as “denial of service” and associated with a particular IPaddress, also see “log” RB 107 in FIG. 3). Other graphical icons 520represent other manners in which a particular RB would cause thegeneration of a satisfied condition object (e.g., upon the particularlog(s), threshold(s) and/or unique value(s) not being observed either aspart of a compound rule with two or more RBs and/or after any desiredtime period or schedule).

The region 524 may generally allow an administrator or other user tomanipulate the one or more RBs making up an AIE rule as well as theirlinking relationships. For instance, the region 524 may includegraphical icons 528 of the RB(s) making up a particular AIE rule inaddition to graphical icons 532 representing the particular linkingrelationship objects connecting adjacent RBs. Numerous manners ofmanipulating the graphical icons 528 are envisioned such as singleclicking or tapping (e.g., with a mouse and cursor or via a touchsensitive screen) on a graphical icon 528 and dragging or deleting thegraphical icon 528, double clicking or tapping on the graphical icon 528to display the user interface 600 (shown in FIG. 12) for configuring theparticular RB, right clicking or tapping on the graphical icon 528 tochange the type of RB (e.g., from a “Log Observed” RB to a “ThresholdNot Observed Scheduled” RB). Similarly, linking relationship objects maybe modified or created by appropriately manipulating the graphical icons532 to display an linking relationship user interface 700 (shown in FIG.20). For instance, appropriate manipulation of the graphical icons 528,532 may bring up pop-up windows, additional user interfaces, and thelike.

Another region 536 of the user interface 500 may be operable to displaysummary type information for RBs and/or their linking relationships. Inone embodiment, manipulating (e.g., single clicking or tapping) agraphical icon 528 or 532 may cause the display of summary informationfor the particular RB or linking relationship corresponding to thegraphical icon 528, 532 in the region 524. In one arrangement, theparticular graphical icon 528, 532 that was manipulated may beappropriately highlighted, marked, or otherwise modified to illustratethat the particular icon 528, 532 is the icon for which summaryinformation is being displayed in region 536 (e.g., by displaying aseries of dots or marks over the icon as shown in FIG. 8). For instance,the top of the region 536 may display a name of the RB or linkingrelationship corresponding to the graphical icon 528, 532 that wasselected (e.g., “Unique Values Observed”), while other portions of theregion 536 may display any specified information for the RB or linkingrelationship such as primary criteria, include filters, etc. (e.g.,components of the filtering module 152 and the like discussed previouslyin relation to FIGS. 3 and 4).

Another region 540 of the user interface 500 may illustrate one or moretime windows within which each of the RBs presented in region 524 willbe processed and/or will evaluate log data, and how such time windowsrelate to the time windows of other RBs. As discussed previously inrelation to FIG. 5, RBs can begin evaluations upon receipt of new data,according to a user-defined or default schedules, in relation to whenother RBs either begin evaluations or determine that their conditionshave been satisfied (e.g., as measured by the time stamps giving rise tothe satisfied condition determination), and the like.

For instance, a toggle bar 544 for one RB (the top toggle bar 544 inFIG. 8) may be set at a relative time of “0” (which represents therelative time that the RB fires or in other words when the RB'scondition was satisfied) and toggle bar 544 for an adjacent RB (seemiddle toggle bar 544 in FIG. 8) may be slid or otherwise manipulatedalong an axis scale 546 so that its left edge lines up with the toptoggle bar 544. In this regard, and regardless of when the first RBbegins processing or evaluation, the second, adjacent RB would in thisexample begin evaluation or processing at or just after the time thatthe first RB's condition was satisfied. In one arrangement, this may bethe default behavior if the linking relationship between the first andsecond RB has an offset of zero.

In other cases, it may be desirable for the second RB to examine factsover the same time period as the first RB. For example, this may beaccomplished by entering a negative offset value in the linkingrelationship properties user interface 700 (discussed below in relationto FIG. 20), which may be accessed by manipulating (e.g., doubleclicking or tapping) the particular toggle bar 544. Similar discussionapplies to the toggle bar 544 representing the third RB (the bottomtoggle bar 544 in FIG. 8). As another example, the toggle bar 544representing a third RB may be dragged so that its left ledge is to theleft of the right edge of the middle toggle bar 544. In this regard, thethird RB may evaluate facts having time stamps associated with timesthat are before the second RB completes an evaluation. Furthermore, thelength of at least some of the toggle bars 544 may represent the totaltime that the particular evaluation of the respective RB will last.

In one arrangement, hovering over a toggle bar 544 will display atooltip 548 (e.g., “hover box”) that identifies the RB, its durationand/or its offset from an adjacent RB. For instance, the tooltip 548 maybe updated to display a current offset while a toggle bar 544 is beingdragged. In another arrangement, an axis scale 546 may be increased asnecessary when dragging toggle bars 544 to the right. In a furtherarrangement, a user may be prevented from dragging a toggle bar 544 offof an axis scale 546 to the left. However, as discussed previously, auser may be able to appropriately bring up the linking relationshipproperties user interface 700 and enter a larger negative offset ifdesired.

Turning now to FIG. 9, another view of the user interface 500 ispresented upon a “settings” tab 508 being manipulated. In general, thesettings tab 508 allows a user to modify one or more types of settingsin relation to a particular AIE rule as a whole. One region 552 of thisview of the user interface 500 may allow for configuration of “commonevent” properties of an AIE rule (i.e., an event generated in responseto all RBs of an AIE rule being satisfied). The region 552 may include aportion that allows a user to define a name for the common event. Forinstance, the user may use the same name as that which defines the AIErule (e.g., “AIE Three Block Rule”) or may define a different name. Theregion 552 may include further portions that allow a classification forthe common event to be defined, such as any of the standardclassifications shared by alarm rules (e.g., “Audit: Startup andShutdown”) and/or a risk rating for the common event.

Another region 556 of the user interface 500 may allow for variousproperties of alarms associated with the AIE rule to be configured. Oneportion of the region 556 may allow a user to opt for an alarm to beautomatically generated (and sent to any appropriate personnel) upongeneration of an event and/or append grouped event field values to analarm notification title. Another portion of the region may allow a userto suppress generated alarms that are the same as a previous alarm for adesired period of time (e.g., so as to avoid the increased networkresource usage involved with generating duplicate alarms). Similarly,and although not shown, another portion of the region may allow a userto suppress generated events that are the same as a previously generatedevent for a desired period of time. Another region 560 of the userinterface 500 may allow a user to specify whether the AIE rule is to beautomatically disabled after a selected date and/or time.

Turning now to FIG. 10, another view of the user interface 500 ispresented upon a “notify” tab 508 being manipulated. In general, thenotify tab 508 allows a user to specify a number of entities that are tobe notified of generated events and/or associated alarms (e.g.,according to role, name or group).

In relation to FIG. 11, another view of the user interface 500 ispresented upon an “information” tab 508 being manipulated. One region564 of this view may allow a user to specify a name of an AIE rule. Asdiscussed previously, the common event name may be the same as (e.g.,via synchronization) the AIE rule name. Another region 568 of this viewmay allow a user to specify a particular group that the AIE rule is tobelong to. The AIE rule can be assigned to an existing group or a newone. Another region 572 may allow a user to enter an optionaldescription of the AIE rule and/or a common event. Another region 576may allow any desired additional details regarding the AIE rule to beentered that are associated with the AIE rule.

As discussed previously, individual RBs can also be configured tospecify which facts the RB is to analyze, filters to be used by the RB,and the like. Turning now to FIG. 12, a user interface 600 is presented(e.g., upon double clicking or tapping on a graphical icon 528 in FIG. 8and/or in one or more other appropriate manners) that allows a user toconfigure a particular RB that makes up an AIE rule. The user interface600 may include a number of tabs 604 that allow a user to toggle betweenvarious screens of the user interface 600 to allow configuration of anRB. One of the tabs may be a “Primary Criteria” tab 604 that allows auser to specify one or more filters, where all of such one or morefilters would have to be met by a fact (e.g., log message or other pieceof data) that is to be collected and/or used by the RB for furtherprocessing (e.g., see previous discussion in relation to primarycriteria filter 160 in FIG. 3). For instance, if primary criteria suchas a particular domain and a particular log message direction (e.g.,inbound or outbound) were specified, then all log messages collectedand/or used by the RB would necessarily be associated with theparticular criteria and particular log message direction. As shown, thisview of the user interface 600 may include at least one portion 608 thatallows a user to add, edit or delete one or more primary criteria. Inone arrangement, at least one primary criteria must be specified foreach RB.

As also shown in FIG. 12, the user interface 600 may also include“Include Filters” and “Exclude Filters” tabs 604 that allow a user tofurther customize an RB. For instance, manipulation of the IncludeFilter tab 604 may present another view of the user interface 600 (notshown) that allows a user to enter or specify one or more filters, wherean incoming fact would need to match at least one of such “includes”filters for that message or piece of data to be collected for furtherprocessing. Manipulation of the Exclude Filter tab 604 may presentanother view of the user interface 600 (not shown) that allows a user toenter or specify one or more filters that may exclude facts if theymatch any of the exclusion filters.

Turning to FIG. 13, another view of the user interface 600 isillustrated after a “Day and Time Criteria” tab 604 has beenmanipulated. This view of the user interface 600 generally allows a userto specify one or more day and/or time intervals within which logmessages or other data must fall within (e.g., as determined by anyappropriate time stamp associated with the logs or data) in order to becollected and/or further processed by the particular RB. As shown inFIG. 13, this view may include an intervals portion 612 whereby a usermay define one or more of such intervals. For instance, each row orinterval may include a number of cells (not labeled) representing startand end days and corresponding start and end times. In one arrangement,the values in a particular cell may be edited by way of manual entry(e.g., clicking or tapping on the cell and using a keyboard to enter adesired value) or via any appropriate drop-down menu.

This view of the user interface 600 may include an add button 620, themanipulation of which adds a new interval to the intervals portion 612.For instance, a first new interval may default to any appropriate dayand time (e.g., Monday, 8 am to 6 pm). Subsequent intervals may defaultto the next day of the week with the same time as the prior interval. Inone embodiment, once Saturday is reached, subsequent intervals maydefault to the next hour of Saturday until midnight is reached (e.g., atwhich point any appropriate error message may be displayed such as“Unable to add filter. The end of the week has been reached”). Inaddition to the add button 620, a delete button 624 may also be includedthat allows a user to remove selected intervals from the intervalsportion 612.

For instance, intervals could be selected by manipulating the particularinterval (e.g., row selectors on the left side of the interval in viewof FIG. 13). As another example, multiple intervals could be selectedusing CTRL and SHIFT. Furthermore, the view may include a time zoneportion 616 that generally allows a user to specify the particular timezone against which the specified intervals will be measured. Forinstance, the time zone may default to a local system time zone whenthis view of the user interface 600 is displayed to a user wheninitially configuring a particular RB. Upon a user changing thedisplayed time zone to a different time zone (e.g., via manipulating adrop down menu), the days and times of the intervals portion 612 mayadjust accordingly.

As also shown in FIG. 13, the user interface 600 may include a “LogSource Criteria” tab 604, the manipulation of which may present anotherview of the user interface 600 (not shown) that allows a user to specifythose particular “facts” (e.g., see above discussion in relation tofacts 124 of FIG. 2) that will be filtered against the aforementionedprimary criteria, include filters, etc.

Turning to FIG. 14, another view of the user interface 600 isillustrated after a “Group By” tab 604 has been manipulated. Asdiscussed previously in relation to FIG. 3, electing to group facts inone or more manners before generation of a satisfied condition object(e.g., satisfied condition object (128 in FIG. 3) allows RB processingto work with totals by type of fact instead of only considering eachfact by itself. In this regard, this view of the user interface 600 mayinclude a portion 628 that allows a user to choose those manners inwhich facts may be grouped (e.g., according to account, classification,common event, impacted country, and the like).

As shown, the portion 628 may include a list 632 of types of grouping,where one or more types may be chosen by selecting a checkbox (althoughnumerous other manners of choosing those manner by which to group factsare envisioned). In any event, any satisfied condition objects generatedby this RB may include metadata with filtering results representing,inter alia, the totals of filtered facts according to the selected“group by” parameter (e.g., see discussion above in relation to FIG. 3).In some embodiments, the details of individual facts (e.g., in relationto the chosen “grouped by” fields) may no longer be available to the RBonce the facts are aggregated according to the selected group byparameters.

With reference to FIG. 15, another embodiment of a user interface 600′is shown. Corresponding components or aspects between the embodiments ofFIGS. 12-14 and 15 are identified by common reference numerals and thosethat differ in at least some respect are identified by a “single prime”designation in FIG. 15. One difference between the user interface 600 ofFIGS. 12-14 and the user interface 600′ of FIG. 15 is the inclusion of a“Thresholds” tab 604′ that is generally operable to allow a user tospecify thresholds for one or more quantitative fields of filtered facts(i.e., facts that have been filtered by any primary criteria, includefilters, exclude filters, etc. as shown in FIG. 12), where meeting orexceeding a particular threshold may cause the generation of a satisfiedcondition object for a particular RB (i.e. may cause the RB to “fire”).

The user interface 600′ may include a portion 636 that allows a user tospecify one or more quantitative fields and corresponding thresholds forconsideration (e.g., see above discussion in relation to FIG. 4). Forinstance, the portion 636 may include a number of cells 640 that allowusers to select one or more quantitative fields (e.g., count, rate,size, etc.) and one or more corresponding thresholds (e.g., via manualentry, drop down menus, etc.). Cells 640 may be added via an add button637 and deleted via a delete button 638 (e.g., after selecting cells fordeletion via appropriately highlighting or marking the particularcell(s)).

Another portion 644 of the user interface 600′ may allow a user tospecify whether a satisfied condition object may be generated eitherupon any of the field/threshold combinations being met, or only upon allfield/threshold combinations being met. Another portion 648 may allow auser to specify an amount of time within which a particular thresholdmust be reached in order for a satisfied condition object to begenerated. For example, if a particular threshold under consideration is100 and the time limit is 15 minutes, then the total of the facts (e.g.,log messages) over any 15 minute time period must reach 100 in order forthe threshold to be reached and a corresponding satisfied conditionobject to possibly be generated (i.e., assuming all other filters havebeen satisfied). As another example, if the time limit is 24 hours andthe RB is only scheduled for 1 hour, then during that hour, the RB willconsider facts generated over the last 24 hours. In one arrangement, ifmultiple thresholds are specified and all must be met, then all must bemet in the same time period.

Turning now to FIG. 16, another embodiment of a user interface 600″ isshown. Corresponding components or aspects between the embodiments ofFIGS. 12-14 and 16 are identified by common reference numerals and thosethat differ in at least some respect are identified by a “double prime”designation in FIG. 16. One difference between the user interface 600 ofFIGS. 12-14 and the user interface 600″ of FIG. 16 is the inclusion of a“Unique Value” tab 604″ that is generally operable to allow a user tospecify a particular number of occurrences of unique values of one ormore particular fields of filtered facts (facts filtered by the primarycriteria, included filters, etc.) that may cause the generation of asatisfied condition object (e.g., more than three unique destination IPaddresses for a given source IP address).

The user interface 600″ may include a portion 652 that allows a user tospecify one or more fields for unique value consideration as well as anumber of occurrences of unique values, the satisfaction of which maycause the generation of a satisfied condition object (see abovediscussion in relation to FIG. 4). For instance, the portion 636 mayinclude a number of cells 654 that allow users to select a field (e.g.,URL, sender, recipient, origin/impacted location, etc.) and acorresponding number of occurrences of unique values of the selectedfield (e.g., via manual entry, drop down menus, etc.). Another portion656 of the user interface 600″ may allow a user to specify a particularperiod of time within which the particular number of occurrences ofunique values of the field must be observed in order for the RB to“fire” (i.e., to generate a satisfied condition object).

With reference to FIG. 17, the user interface 600″ is illustrated afteran “Information” tab 604″ has been manipulated. This view of the userinterface 600″ may include a portion 660 that allows a user to enter adesired description of the particular RB that is being created oredited. In one arrangement, the description entered in portion 660 maybe displayed on a user's display as a tooltip upon the user moving acursor or other component over a corresponding graphical icon 528 in therule block designer region 524 in the user interface 500 of FIG. 8.

Turning now to FIG. 18, another embodiment of a user interface 600′″ isshown. Corresponding components or aspects between the embodiments ofFIGS. 12-14 and 18 are identified by common reference numerals and thosethat differ in at least some respect are identified by a “triple prime”designation in FIG. 18. One difference between the user interface 600 ofFIGS. 12-14 and the user interface 600′″ of FIG. 18 is the inclusion ofa “Schedule” tab 604′″ that is generally operable to allow a user tospecify one or more time intervals within which the particular RB willbe evaluated. This feature may be useful when it is desired to monitorfor the non-occurrence of a particular occurrence (e.g., determiningwhether a “server backup completion” log message has not been observedafter a server backup has begun). Stated otherwise, this feature may beused to evaluate a particular RB during those times when factsindicative of particular occurrences would normally be expected to begenerated. Thus, in the absence of receipt of a log message indicativeof such an occurrence, the RB may fire.

The user interface 600′″ may include a portion 664 that allows a user tospecify whether the non-occurrence or non-observed nature of aparticular occurrence is to be always active (i.e., the RB is to becontinuously evaluated) or active according to a customized schedule(and, if so, a particular local time zone to which a normalized time ofreceived log messages will be converted before being compared with thecustomized schedule to determine if they should be considered orevaluated by the RB). In one arrangement, a user may select to utilizedaylight savings time rules when converting normal message times tolocal times.

In any event, the portion 664 may include a number of cells 668 thatallow users to specify one or more time intervals within which the RBwill be evaluated (e.g., by start and end days and respective start andend times). Cells 668 may be added via an add button 669 and deleted viaa delete button 670 (e.g., after selecting cells for deletion viaappropriately highlighting or marking the particular cell(s)). In onearrangement, a first monitoring interval may be from 8 am to 5:59 pm onMonday by default, subsequent intervals may use the same time forTuesday through Saturday, and added intervals may advance by one houruntil the end of the week is reached. Numerous other schedules areenvisioned.

The user interface 600′″ may also include a portion 672 that allows auser to specify an evaluation frequency for RB evaluation. For instance,the RB may be initially evaluated starting at the beginning of each ofthe time intervals shown in cells 668, and proceeding for the evaluationfrequency amount of time specified in portion 672. Upon thenon-observance of a fact representative of a particular occurrence(e.g., a log message indicative of a completion of a server backup) atthe end of the first evaluation frequency period of time, the RB mayfire a pending event message. The RB would continue to determine foreach subsequent evaluation frequency period of time during the timeinterval whether an occurrence was “not observed,” and, if so notobserved, generate a satisfied condition object at the end of eachevaluation frequency period of time. In one arrangement, a user may beable to suppress the generation of identical satisfied condition objectsduring any appropriate period of time.

With reference now to FIG. 19, another view of the user interface 600′″is presented after a “Distinct” tab 604′″ has been manipulated. This tabis generally operable to allow a user to specify a list of entities(e.g., hosts) for which a fact indicative of some occurrence (e.g.,backup succeeded) should be observed. This view of the user interface600′″ may include a portion 676 that allows a user to select a field forwhich to specify distinct occurrences (e.g., via a drop-down menu).Another portion 680 may be included that allows a user to specify one ormore values for the field for which the RB should monitor.

It should be appreciated that at least some of the above discussed userinterfaces 600, 600′, 600″ and 600′″ or other similar user interfacesmay be utilized to create one or more of “observed” RBs (i.e., RBs thatfire upon a particular type of log(s), threshold(s) or unique value(s)being observed), “not observed scheduled” RBs (i.e., RBs that fire upona particular type of log(s), threshold(s) or unique value(s) not beingobserved according to some particular schedule and/or evaluationfrequency) and “not observed compound” RBs (i.e., RBs having at leastone preceding RB that fire upon a particular type of log(s),threshold(s) or unique value(s) not being observed). For instance, whilethe user interfaces 600, 600′ were respectively discussed above in thecontext of a RB firing upon a particular type or types of log(s) andquantitative field threshold(s) being observed, the same user interfaces600, 600′ or other user interfaces may be utilized to generate “notobserved scheduled” log and/or threshold RBs and/or “not observedcompound” log and/or threshold RBs. Numerous other combinations andexamples of RBs are envisioned and are within the scope of the presentdisclosure.

FIG. 20 illustrates a user interface 700 that allows a user to configurea linking relationship object (e.g., linking relationship object 182 ofFIG. 3) for a particular RB. As discussed previously, one manner ofaccessing the user interface 700 may be by manipulating (e.g., doubleclicking or tapping) a corresponding linking relationship graphical icon532 illustrated in FIG. 8. As shown, the user interface 700 may includea portion 704 that allows a user to specify at least one common “groupby” field (see above discussion of group by fields in relation to FIG.3) between facts of a current RB (e.g., “Rule Block 3” in FIG. 20) andfacts of an adjacent RB (e.g., “Rule Block 2” in FIG. 20).

The portion 704 may include respective cells 708 for one RB group byfield, an adjacent RB group by field, and an operator (e.g., equals, or)that links the current and preceding RB group by fields. For instance,upon Rule Block 3 in FIG. 20 firing first, the specific content of the“classification” field (e.g., “Reconnaissance”) of the fact(s) whichcaused Rule Block 3 to fire would be used as a key into an index forfacts of Rule Block 2 to determine if Rule Block 2′s condition has beensatisfied with respect to the specific key. In one arrangement, onlythose fields that were utilized to “group” received and processed factsat a particular RB may be available for selection of a group by field ina linking relationship object. As an example, a user may configure theadjacent RB group by fields by manipulating a drop-down menu on acorresponding cell 708 which may be populated by a list of onlyavailable group by fields. In another arrangement, the list for one RBmay be populated with all group by fields in the RB which are compatiblewith a selected field for the adjacent RB. For example, if the selectedgroup by field for the field of one RB is an IP address, then it canonly be related to an IP address in the adjacent RB. That is, it wouldnot make sense to try to relate an IP address to a common event becausethey identify different things.

The user interface 700 may also include a portion 712 that allows a userto specify a particular period of time within which the particular RBmust be satisfied. This period of time may also be set or modified viaone or more of the user interfaces 600, 600′ 600″, 600′″. Anotherportion 716 allows the user to set or modify an offset of a time windowprocessed by a particular RB relative to a time window of a previous RB(e.g., see previous discussion in relation to FIG. 5). Either or both ofthe information of portions 712 and 716 may be encapsulated with theabove-discussed linking relationship objects.

Turning now to FIG. 21, a screenshot of a user interface 800 ispresented that allows a user to observe and investigate on a number of“AIE events” (i.e., events generated via AIE processing). For instance,the user interface 800 may include a number of rows and columns of cells804, where each row represents an event and the cells of each columninclude metadata (e.g., log source entity, classification, direction,etc.) making up each of the events. To obtain more detailed informationregarding a particular event, a user may “drill-down” into or otherwisemanipulate an event (e.g., such as event 808) in any appropriate manner.With respect to FIG. 22, a screenshot of a user interface 900 isillustrated (e.g., after a user has drilled down into event 808) thatgenerally shows the number of facts (e.g., logs) that satisfied theconditions of each of the one or more RBs (as shown, RB#1 and RB#2) ofthe particular AIE rule and that lead to the generation of event 808.

For instance, the user interface 900 may include a number of rows andcolumns of cells 904, where each row represents an RB and the cells ofeach column include metadata (e.g., log manager name, status, log count,etc.) regarding each of the RBs. Upon appropriately manipulating theuser interface 900 (e.g., clicking “ok” button 908), the user may bepresented with a screenshot of another user interface 1000 asillustrated in FIG. 23. As shown, the user interface 1000 generallyconveys how the pertinent facts for each RB (i.e., those facts that leadto the satisfaction of the RB's condition as discussed previously) havebeen grouped together for efficient viewing. For instance, the userinterface 1000 may include a number of rows and columns of cells 1004,where each row represents an aggregate fact (e.g., log) and each cell ofthe columns includes metadata (e.g., log source entity, classification,direction, etc.) making up each aggregate fact. The user interface 1000may include collapse/expand buttons 1008 for each RB to allow a user toselectively show or hide the various facts that lead to the satisfactionof the condition of each RB. In the illustrated example, the group of 43aggregate facts for RB#1 has been collapsed while the single factmeeting criteria for RB#2 has been displayed.

FIG. 25 presents a flow diagram or protocol 1200 for use in monitoringone or more platforms of one or more data systems (e.g., devices 22 ornetworks 26 in FIG. 1). The protocol 1200 may include receivinglog-related data (e.g., facts 124 in FIG. 2) at a processing engine(e.g., at AIE 100 in FIG. 2) at 1204, and evaluating at least some ofthe data using a first rule block (e.g., RB₁ 108 in FIG. 2) at 1208 todetermine whether a result of the evaluation is a first outcome (e.g.,an interesting occurrence or development) or a second outcome (e.g., anon-interesting occurrence or development) at 1212. For instance, thefirst outcome may be that a particular log(s) or quantitativethreshold(s) has been observed. In response to the result being thesecond outcome, the protocol 1200 may flow back to 1204 wherebyadditional log-related data may be received and/or 1208 where the firstRB may again be evaluated.

In response to a determination that the result is the first outcome at1212, a satisfied condition object may be generated for the first RB andthe protocol 1200 may flow to 1216 whereby a second RB (e.g., RB₂ 112 inFIG. 2) may be evaluated to determine whether a result of the evaluationis a first (interesting/desired) outcome or a second(non-interesting/non-desired) outcome at 1220. In response to the resultbeing the second outcome, the protocol 1200 may flow back to 1204whereby additional log-related data may be received, 1208 where thefirst RB may again be evaluated, or 1216 where the second RB may againbe evaluated. In response to a determination that the result is a firstoutcome at 1220, a satisfied condition object may be generated for thesecond RB and an event may then be generated at 1224. It should beappreciated that numerous modifications and/or additions to the protocol1200 are envisioned as well as other protocols for use with or by theteachings presented herein.

It should also be appreciated that the various user interfaces disclosedherein are only presented for exemplary purposes and should not be seenas limiting the present disclosure in any regard. Rather, the userinterfaces are only meant to assist the reader in understanding onemanner of implementing the teachings disclosed herein on one or morecomputer systems. Similarly, the various manners of access (e.g., rightclicking on a line to bring up a particular drop down menu, and thenselecting a particular tab or button, etc.) to the various modules andfunctionalities (e.g., creating RBs, specifying linking relationshipobjects) disclosed herein should not be seen as limiting such tools andfunctionalities to the particular manners of access discussed or evenlimiting such modules and functionalities to use with any manners ofaccess at all. Rather, these manners of access have only been shown asexamples to assist the reader in understanding how one may access suchmodules on a use interface. Other manners of access to the variousmodules and functionalities are also envisioned and encompassed withinthe scope of the various embodiments.

Furthermore, embodiments disclosed herein can be implemented as one ormore computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer-readable medium for executionby, or to control the operation of, data processing apparatus. Forexample, the various modules and managers utilized by the system 10 ofFIG. 1 to process facts, generate events, take remedial action, and thelike, may be provided in such computer-readable medium and executed by aprocessor or the like. The computer-readable medium can be amachine-readable storage device, a machine-readable storage substrate, amemory device, a composition of matter affecting a machine-readablepropagated signal, or a combination of one or more of them. The system10 may encompass one or more apparatuses, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. In addition to hardware,the system 10 may include code that creates an execution environment forthe computer program in question, e.g., code that constitutes processorfirmware, a protocol stack, a database management system, an operatingsystem, or a combination of one or more of them.

A computer program (also known as a program, software, softwareapplication, script, or code) used to provide the functionalitydescribed herein (such as to provide the various artifact rightsmanagement processes disclosed herein) can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program does not necessarilycorrespond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub-programs, or portionsof code). A computer program can be deployed to be executed on onecomputer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read-only memory or a random access memoryor both. Generally, the elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. The techniques described herein may beimplemented by a computer system configured to provide the functionalitydescribed.

In different embodiments, system 10 (i.e., the one or more log managers30, AIEs 50, event managers 38, and the like) may include one or more ofvarious types of devices, including, but not limited to a personalcomputer system, desktop computer, laptop, notebook, or netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, application server, storage device, a consumerelectronics device such as a camera, camcorder, set top box, mobiledevice, video game console, handheld video game device, a peripheraldevice such as a switch, modem, router, or, in general, any type ofcomputing or electronic device.

Typically, a computer will also include, or be operatively coupled toreceive data from or transfer data to, or both, one or more mass storagedevices for storing data, e.g., magnetic, magneto-optical disks, oroptical disks. However, a computer need not have such devices. Moreover,a computer can be embedded in another device, e.g., a mobile telephone,a personal digital assistant (PDA), a mobile audio player, a GlobalPositioning System (GPS) receiver, a digital camera, to name just a few.Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry. Toprovide for interaction with a user (e.g., via the user interfacesdiscussed herein), embodiments of the subject matter described in thisspecification can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor,for displaying information to the user and a keyboard and a pointingdevice, e.g., a mouse or a trackball, by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

While this disclosure contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular embodiments of the disclosure. Certain features that aredescribed in this specification in the context of separate embodimentscan also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Additionally, the foregoing description of the present invention hasbeen presented for purposes of illustration and description.Furthermore, the description is not intended to limit the invention tothe form disclosed herein. Consequently, variations and modificationscommensurate with the above teachings, and skill and knowledge of therelevant art, are within the scope of the present invention. Theembodiments described hereinabove are further intended to explain bestmodes known of practicing the invention and to enable others skilled inthe art to utilize the invention in such, or other embodiments and withvarious modifications required by the particular application(s) oruse(s) of the present invention. It is intended that the appended claimsbe construed to include alternative embodiments to the extent permittedby the prior art.

What is claimed is:
 1. A method for use in monitoring one or moreplatforms of one or more data systems, comprising: first evaluating, bya processor using a first rule block, structured data received from oneor more platforms over at least one communications network; firstdetermining, from the first evaluating, that a result is one of at leastfirst and second outcomes; accessing, by the processor, a linkingrelationship object in the first rule block to identify a data field inthe structured data; extracting, by the processor, a content of the datafield from the structured data; second evaluating, by the processorusing a second rule block, structured data associated with the extractedcontent received from the one or more platforms; second determining,from the second evaluating, whether a result is one of at least firstand second outcomes; and analyzing the results of the first and seconddetermining to determine an event of interest.
 2. The method of claim 1,further comprising: passing the extracted content to the second ruleblock, wherein the second evaluating is performed in response to receiptof the extracted content by the second rule block.
 3. The method ofclaim 1, wherein the second evaluating comprises: searching a pluralityof entries of an indexing structure of the second rule block for anentry that corresponds to the extracted content.
 4. The method of claim3, wherein the second evaluating comprises: filtering, by the processorusing the second rule block, the structured data to generate second ruleblock filtered data, wherein each entry of the plurality of entries ofthe indexing structure of the second rule block indicates a presence ofa corresponding portion of the second rule block filtered data.
 5. Themethod of claim 4, wherein the filtering occurs before the firstdetermining.
 6. The method of claim 4, further including: determiningthat an entry exists in the plurality of entries of the indexingstructure of the second rule block that corresponds to the extractedcontent; obtaining the portion of the second rule block filtered datathat corresponds to the determined entry; and generating a satisfiedcondition object based on the obtained portion of the second rule blockfiltered data, wherein the analyzing includes analyzing the satisfiedcondition object of the second rule block.
 7. The method of claim 4,further comprising: accessing, by the processor, a linking relationshipobject in the second rule block to identify a data field; and generatingthe indexing structure of the second rule block based on the identifieddata field from the linking relationship object of the second ruleblock.
 8. The method of claim 7, wherein the second rule block filtereddata includes a plurality of portions of structured data, and furthercomprising: obtaining, by the processor, a content of the identifieddata field for each of the plurality of structured data portions,wherein the generating includes using the content of the identified datafield from the linking relationship object of the second rule block fromeach of the plurality of structured data portions of the second ruleblock filtered data to generate the indexing structure of the secondrule block.
 9. The method of claim 7, wherein the first evaluatingcomprises: filtering, by the processor using the first rule block, theat least some of the data to generate first rule block filtered data;and generating an indexing structure of the first rule block based onthe identified data field from the linking relationship object of thefirst rule block, wherein the indexing structure of the first rule blockincludes a plurality of entries, and wherein each entry of the pluralityof entries of the indexing structure of the first rule block indicates apresence of a corresponding portion of the first rule block filtereddata.
 10. The method of claim 9, wherein the identified data field fromthe linking relationship object of the first rule block is differentthan the identified data field from the linking relationship object ofthe second rule block.
 11. The method of claim 9, wherein the first ruleblock filtered data includes a plurality of portions of structured data,and further comprising: obtaining, by the processor, the content of theidentified data field for each of the plurality of structured dataportions, wherein the generating includes using the content of theidentified data field from the linking relationship object of the firstrule block from each of the plurality of structured data portions of thefirst rule block filtered data to create the indexing structure of thefirst rule block.
 12. The method of claim 3, wherein the result of thesecond determining corresponds to data associated with the extractedcontent and evaluated by the second rule block.
 13. A non-transitory,computer-readable storage medium, storing program instructions that whenexecuted on one or more computers cause the one or more computers toperform: first evaluating, using a first rule block, structured datareceived from one or more platforms over at least one communicationsnetwork; first determining, from the first evaluating, that a result isone of at least first and second outcomes; accessing a linkingrelationship object in the first rule block to identify a data field inthe structured data; extracting a content of the data field from thestructured data; second evaluating, using a second rule block,structured data associated with the extracted content received from theone or more platforms; second determining, from the second evaluating,whether a result is one of at least first and second outcomes; andanalyzing the results of the first and second determining to determinean event of interest.
 14. The non-transitory, computer-readable storagemedium of claim 13, further storing program instructions that whenexecuted on one or more computers cause the one or more computers toperform: passing the extracted content to the second rule block, whereinthe second evaluating is performed in response to receipt of theextracted content by the second rule block.
 15. The non-transitory,computer-readable storage medium of claim 13, wherein the secondevaluating comprises: searching a plurality of entries of an indexingstructure of the second rule block for an entry that corresponds to theextracted content.
 16. The non-transitory, computer-readable storagemedium of claim 15, wherein the second evaluating comprises: filtering,using the second rule block, the structured data to generate second ruleblock filtered data, wherein each entry of the plurality of entries ofthe indexing structure of the second rule block indicates a presence ofa corresponding portion of the second rule block filtered data.
 17. Thenon-transitory, computer-readable storage medium of claim 16, whereinthe filtering occurs before the first determining.
 18. Thenon-transitory, computer-readable storage medium of claim 16, furtherstoring program instructions that when executed on one or more computerscause the one or more computers to perform: determining that an entryexists in the plurality of entries of the indexing structure of thesecond rule block that corresponds to the extracted content; obtainingthe portion of the second rule block filtered data that corresponds tothe determined entry; and generating a satisfied condition object basedon the obtained portion of the second rule block filtered data, whereinthe analyzing includes analyzing the satisfied condition object of thesecond rule block.
 19. The non-transitory, computer-readable storagemedium of claim 16, further comprising: accessing a linking relationshipobject in the second rule block to identify a data field; and generatingthe indexing structure of the second rule block based on the identifieddata field from the linking relationship object of the second ruleblock.
 20. The non-transitory, computer-readable storage medium of claim19, wherein the second rule block filtered data includes a plurality ofportions of structured data, and further comprising: obtaining a contentof the identified data field for each of the plurality of structureddata portions, wherein the generating includes using the content of theidentified data field from the linking relationship object of the secondrule block from each of the plurality of structured data portions of thesecond rule block filtered data to generate the indexing structure ofthe second rule block.
 21. The non-transitory, computer-readable storagemedium of claim 19, wherein the first evaluating comprises: filtering,using the first rule block, the at least some of the data to generatefirst rule block filtered data; generating an indexing structure of thefirst rule block based on the identified data field from the linkingrelationship object of the first rule block, wherein the indexingstructure of the first rule block includes a plurality of entries, andwherein each entry of the plurality of entries of the indexing structureof the first rule block indicates a presence of a corresponding portionof the first rule block filtered data.
 22. The non-transitory,computer-readable storage medium of claim 21, wherein the identifieddata field from the linking relationship object of the first rule blockis different than the identified data field from the linkingrelationship object of the second rule block.
 23. The non-transitory,computer-readable storage medium of claim 21, wherein the first ruleblock filtered data includes a plurality of portions of structured data,and further storing program instructions that when executed on one ormore computers cause the one or more computers to perform: obtaining thecontent of the identified data field for each of the plurality ofstructured data portions, wherein the generating includes using thecontent of the identified data field from the linking relationshipobject of the first rule block from each of the plurality of structureddata portions of the first rule block filtered data to create theindexing structure of the first rule block.
 24. The non-transitory,computer-readable storage medium of claim 15, wherein the result of thesecond determining corresponds to data associated with the extractedcontent and evaluated by the second rule block.